DEV Community: ULNIT

My AI Agent Ran a 3-Hour Task Perfectly, Then Emailed 40 Customers About Work That Wasn't Done

ULNIT — Thu, 17 Sep 2026 01:03:53 +0000

I run a support-and-ops agent on a Raspberry Pi that handles long, multi-step jobs: triaging inboxes, reconciling order webhooks, writing the daily summary. For weeks it was flawless on anything under an hour. Then one Tuesday it spent three hours on a migration task, and in the final step it did the one thing I had explicitly, in bold, at the top of its instructions, told it never to do.

It emailed 40 customers about a change that hadn't shipped yet.

No tool bug. No hallucinated fact. No prompt regression. The rule was still sitting in my system prompt, exactly where I'd written it. The problem was that by hour three, the model could no longer see it — because the runtime had quietly summarized the first 80% of the conversation away to fit the context window, and summaries, it turns out, are lossy in the worst possible direction: they keep the recent and the loud, and drop the quiet constraints stated once, long ago.

This is a post-mortem of that failure and the architecture that replaced it. If you run agents on tasks longer than a single context window, you will hit this. I'd rather you hit it reading this than hitting it on your customer list.

What actually happened

My agent loop was bog-standard: system prompt + task, then tool calls and results accumulating in one conversation until done. The migration task involved ~200 tool calls — file reads, dry-run outputs, a database diff. Around call 140, my provider's context management kicked in and compacted the older turns into a summary so the run could continue.

Three things got lost in that compaction:

The constraint. "Do NOT send any customer emails until the migration is verified and I approve manually" was stated once, in the system prompt, ~3 hours and 100k tokens earlier. The compaction treated the whole conversation as one blob. Early instructions got folded into a two-paragraph summary that captured what the agent was doing but not what it was forbidden from doing.
The verification step. The plan had a "verify checksums, then wait for approval" gate. After compaction, the agent's working memory said something like "migration mostly complete, remaining step: notify customers." The gate wasn't remembered as a gate — it was remembered as, at best, a suggestion.
Its own uncertainty. Before compaction, the agent had noted "diff shows 3 unresolved conflicts." That note got summarized into "migration proceeding normally." This is the scariest loss: the summary didn't just drop detail, it dropped doubt, and doubt is what makes agents check before acting.

So the agent finished the last file operation, saw "notify customers" as the only remaining step in its compressed memory, and cheerfully sent 40 emails about a migration that was in a broken intermediate state. I caught it in 20 minutes and spent the next day sending corrections. Reputation cost: real but survivable. Nerve cost: significant.

The naive fixes, and why they failed

Before I got to the real fix, I tried two obvious things. Both failed in instructive ways.

Fix attempt #1: LOUDER PROMPT. I moved the constraint to the top, capitalized it, added "CRITICAL" and "NEVER." It survived slightly longer into the run — and then got compacted anyway, because compaction doesn't care about your font choices. It cares about token position and recency. If your safety rule lives only in text that will eventually be summarized, it has an expiration date. Lesson: emphasis is not persistence.

Fix attempt #2: just use a bigger context window. I switched to a model with 4x the context. This "worked" for two weeks and then failed the same way on a bigger task — and every run got slower and ~3x more expensive. Lesson: a bigger window doesn't fix the architecture, it just moves the cliff. Any fixed window eventually meets a longer task. And if your agent's memory is the conversation itself, every failure mode of conversations — drift, dilution, compaction — becomes a failure mode of your agent's state.

The fix: state lives on disk, not in the conversation

The mental shift that solved this: the conversation is a scratchpad, not the memory of record. Anything that must survive hour three has to live outside the context window, in a place the agent re-reads on every loop.

I now run long tasks with three files:

1. constitution.md — constraints that are re-injected every loop. Not appended to history: literally prepended to the model input at the start of every iteration, so they're always in the most recent, never-compacted region of the window. Ten to fifteen lines max, or it stops being a constitution and becomes another document the model skims.

# INVARIANT RULES (never summarizable, always current)
- No customer-facing sends without explicit approval flag in state.json
- No money movement above $5 without approval flag
- If state.json says verified=false, the task is NOT complete
- When unsure whether a step is allowed: stop and write the question to state.json

2. state.json — the single source of truth about task progress. The agent updates it after every meaningful step, and reads it before every decision. Compaction can shred the conversation; it can't touch the file.

{
  "task": "orders_migration_sept",
  "phase": "migrated_unverified",
  "verified": false,
  "approval": null,
  "open_conflicts": ["row 1184", "row 2210", "row 3977"],
  "next_action": "resolve conflicts, run checksum verify",
  "notes_for_future_self": "diff tool truncates at 50 rows — re-run with --full"
}

Two fields did most of the work: verified (a boolean gate the constitution references by name, so losing the prose doesn't lose the rule) and notes_for_future_self, which is where the agent writes down its doubts. That last field directly addresses failure #3 above — after compaction the agent no longer remembers being uncertain, but it can read that it was uncertain.

3. A checkpoint ritual. Every 30 minutes of wall-clock time, the agent appends a summary of what it did and what changed to log.md, on disk. If the run crashes, gets compacted badly, or I kill it and restart, the fresh agent reads constitution.md + state.json + the tail of log.md and resumes in the right mental state within one tool call. The conversation history becomes disposable — which is exactly what you want, because it was never reliable in the first place.

The loop looks like this:

while not done:
    prompt = (
        read("constitution.md")          # always fresh, never compacted
        + "\n\nCURRENT STATE:\n" + read("state.json")
        + "\n\nRECENT LOG:\n" + tail("log.md", 40)
        + "\n\nContinue the task."
    )
    result = run_agent_turn(prompt, tools)
    # agent's tool calls update state.json / log.md on disk

Note what's not in there: the full conversation. Each turn is nearly stateless. The model gets the rules, the truth, and the recent past — reconstituted from disk every time. Compaction stopped mattering because there was nothing left worth compacting.

What it cost, and what I'd do differently

Honest accounting, because the setup isn't free:

More tool calls per turn. Reading three files every loop adds latency and tokens. In practice it made runs cheaper overall, because I stopped replaying 100k-token conversations and the agent stopped redoing work it had forgotten completing.
The agent sometimes writes bad state. Early on it would mark "verified": true optimistically. I fixed that the boring way: the verify step is a script, not an agent judgment call, and only the script writes the verified field. The agent can request verification; it can't grant it. If you take one thing from this post, take this: make critical state transitions the output of deterministic code, not model opinion.
I should have started here. In hindsight, the three-file pattern is not a "fix" I bolted on after a failure — it's the minimum viable architecture for any agent task longer than ~an hour, and I should have assumed day one that every long run eventually gets compacted, truncated, or restarted. Design for an agent with amnesia that can read its own notebook, not an agent that remembers.

The 40 customers got a correction email within a day, and three of them replied with some version of "happens, your bot was at least honest about it." I'll take it.

The deeper point generalizes well past this one bug: your agent's reliability is bounded by wherever its state lives. If state lives in the conversation, it inherits every weakness of the conversation. Put the rules and the truth on disk, re-read them every loop, and let the model forget everything else — it was only ever a scratchpad anyway.

I write up the specific playbooks in The Solo Operator's AI Agent Playbook — code LAUNCH90 at checkout makes it $1.90. If it doesn't save you 5 hours in week one, reply to the receipt for a refund.

I Replaced My Agent's 900-Word Instruction Prompt With 6 Examples. It Started Confidently Selling a Product I Retired in 2024.

ULNIT — Wed, 16 Sep 2026 01:03:18 +0000

My support agent ran on a 900-word instruction prompt that I'd been polishing for two months. Every time it made a mistake, I added another rule. "Never promise refunds above $50 without flagging." "If the customer is angry, acknowledge first." "When unsure about shipping times, say 5–7 business days, not 3–5."

Nine hundred words of rules, and the agent still occasionally sounded like a lawyer apologizing for existing.

So in July I tried the thing every prompt engineering article tells you to do: I deleted most of the instructions and replaced them with six hand-written examples of ideal responses. Few-shot prompting. Supposedly a slam dunk.

It got worse immediately — in a way I didn't understand for about a week. This post is what I learned: why examples beat instructions most of the time, the specific failure mode nobody warned me about, and the structure I ended up with.

Why examples beat instructions (when they work)

Instructions describe the target. Examples are the target. A model doesn't have to interpret "be concise but warm" — those words map to a fuzzy region of behavior space. But a concrete example of a concise-but-warm reply pins the tone, length, and format exactly, with zero interpretation overhead.

Three things improved the day I switched:

1. Format compliance went from ~85% to ~99%. I'd been begging the agent (in prose) to always end shipping questions with the tracking-link pattern. One example that ended that way did more than three paragraphs of rules.

2. Tone stopped drifting. With instructions, tone depended on which rules the model happened to weight on a given run. With examples, the tone was anchored. Every output sounded like it came from the same human.

3. The prompt got shorter and cheaper. Six examples plus 120 words of framing cost fewer tokens than my 900-word rulebook, and I could finally read the whole prompt in one screen.

If that were the whole story, this would be a boring post.

The failure: my agent started hallucinating my examples

Here's what nobody told me. On day three, a customer asked about a product I don't even sell anymore — an old plan called "Starter Tier" that I'd retired in 2024. One of my six examples happened to mention Starter Tier pricing, because I'd written the examples from real (but old) support threads.

The agent replied, confidently, with the retired plan's price. The customer tried to buy it. I got an email asking why checkout didn't work.

That was the visible failure. The invisible one was worse: I checked the logs and found the agent had been pattern-matching my examples as facts for three days. Any question that was even loosely near an example got answered with details from the example — including numbers, timeframes, and policy specifics that were frozen in whatever moment I'd written them.

I had accidentally created a tiny, authoritative-looking knowledge base of stale facts and told the model "responses look like this." The model heard "these are true things."

The root cause, once I saw it: examples carry two kinds of information — style and content — and the model can't tell which parts you mean as demonstration and which as ground truth.

The fix: separate style from facts

I restructured the prompt into three explicit layers, and this is the part I'd hand anyone starting out:

LAYER 1 — FACTS (the only source of truth)
Current plans, prices, shipping windows, refund policy.
"Only use information from this section for any specific
number, date, price, or policy. If it's not here, say you'll
check and escalate."

LAYER 2 — STYLE EXAMPLES (demonstrations, not facts)
3–5 example exchanges. Prefaced with:
"These examples demonstrate TONE, LENGTH, and STRUCTURE only.
The prices, plans, and details in them may be fictional.
Never copy specific facts from these examples into replies."

LAYER 3 — ESCALATION RULES (short)
5–10 bullet rules for the genuinely hard cases: angry
customers, refund requests over $X, anything legal.

Two details that mattered more than I expected:

I deliberately made the style examples slightly fictional. Changed names, rounded numbers, invented order IDs. If the agent ever leaked example content into a real reply, it would be obviously wrong ("Order #12345") instead of subtly wrong (a real-looking but outdated price). Subtle wrongness is what gets you a refund dispute. Obvious wrongness gets you a caught bug.

I added one anti-example. A single "BAD response" with one line explaining why it's bad — in my case, an over-apologetic three-paragraph reply. Negative examples are underrated; they draw the boundary of the style region from the other side, and they cost almost nothing.

After the restructure, format compliance stayed at ~99%, tone stayed anchored, and fact hallucinations from examples went to zero in the following eight weeks. Not because the model got smarter — because the prompt finally told it which parts were a demonstration.

Rules I now follow when writing example-driven prompts

Examples for style, explicit data for facts, never mix them. If an example contains a real number, that number will eventually be repeated to someone who shouldn't hear it.
Audit examples like you'd audit dependencies. Every example is a frozen snapshot of your business. When prices change, examples go stale silently. I re-read mine on the first of every month — calendar invite, ten minutes, non-negotiable.
Three to six examples is the sweet spot. Below three, the style isn't pinned. Above six, they start contradicting each other in subtle ways and you're paying tokens for noise.
Include at least one hard case. Don't make all your examples happy-path. One example where the correct answer is "let me check and get back to you" teaches the agent that not-knowing is an acceptable output — that single example probably prevented more damage than everything else combined.
Test with adversarial inputs before shipping. I threw 20 weird real customer emails at the new prompt, including ones near the edges of my examples. That's how I'd have caught the Starter Tier problem on day one instead of day three.

What I'd tell myself two months ago

Instructions aren't dead — my Layer 3 rules are instructions, and they're load-bearing. The lesson isn't "examples > rules." It's that the two do different jobs, and most broken agent prompts I've seen (including mine) are broken because one is doing the other's job. Rules trying to describe tone produce lawyer-speak. Examples trying to carry facts produce confident lies.

Give each layer its job, label the layers explicitly, and the model does what you meant instead of what you wrote.

All 100 prompts are in The Agent Prompt Vault — $3, lifetime updates. Steal the ones that fit your workflow.

My Raspberry Pi's SD Card Died at 2 AM. My "Automatic" Backups Turned Out to Be a 47-Day-Old Lie.

ULNIT — Tue, 15 Sep 2026 01:03:20 +0000

The alert came in at 2:14 AM: three of my automation agents had missed their heartbeat window. By the time I SSHed in from my phone, I already knew — the Pi that runs my entire one-person business stack had stopped responding mid-write. When I finally pulled the power and booted from a fresh card, fsck gave me the verdict: the filesystem was corrupt beyond repair.

No problem, I thought. I have nightly backups. I've had them since day one.

The backup ran every night at 3 AM. It logged success every single night. And when I opened the backup directory, the newest snapshot was 47 days old.

This is the post-mortem of how a backup system that "worked" for a month and a half was actually broken from roughly week one — and the handful of cheap, boring fixes that made sure it can never lie to me again.

What the stack was

For context, this Pi runs the unglamorous core of my setup:

Two AI agents (support triage and a content scheduler) running as systemd services
A small Python job runner with cron-triggered tasks
An SQLite database that holds customer state, agent memory, and job history
A reverse proxy in front of two internal web apps

Nothing exotic. Total cost: a $60 Pi, one SD card, and the assumption that "I set up backups" means "I have backups."

The backup that wasn't

My backup was a shell script in /etc/cron.daily. Roughly this:

#!/bin/bash
rsync -a --delete /home/sean/agents/ /mnt/usb/backup/agents/
sqlite3 /home/sean/agents/state.db ".backup /mnt/usb/backup/state.db"
echo "$(date): backup OK" >> /var/log/backup.log

It looked correct. It even logged success. Three separate bugs were stacked on top of each other:

Bug 1: The USB drive had silently unmounted. About 47 days before the crash, the drive dropped off the bus after a brief power dip. The mount point /mnt/usb still existed — as an empty directory on the SD card. rsync happily wrote into it, consuming SD card space instead of backing anything up.

Bug 2: The log line always said OK. The echo ran unconditionally. I never checked $? on the rsync or the sqlite backup. "backup OK" was printed whether or not anything had been copied. I built a system whose only job was to tell me the truth, and I wrote it so it could only tell me one thing.

Bug 3: Nobody read the log anyway. Even a correct log line goes nowhere if the only failure mode is "human remembers to check." I had alerting on agent heartbeats but zero alerting on the thing whose entire purpose was disaster recovery.

The irony: I monitor my AI agents obsessively. I got paged when an agent missed a heartbeat by six minutes. But the backup script — the thing standing between me and total data loss — could fail silently for seven straight weeks and I'd never know.

The fix: backups must prove themselves

I rebuilt the backup around one principle: a backup that doesn't verify itself is a rumor, not a backup. Here's the current version, trimmed:

#!/bin/bash
set -euo pipefail

DEST=/mnt/usb/backup
STAMP=$(date +%F)
FAIL=0

# 1. Prove the destination is real storage, not an empty mountpoint
if ! mountpoint -q /mnt/usb; then
  notify "BACKUP FAIL: /mnt/usb not mounted"
  exit 1
fi

# 2. Prove there's space
AVAIL=$(df --output=avail -BM /mnt/usb | tail -1 | tr -dc '0-9')
if [ "$AVAIL" -lt 2000 ]; then
  notify "BACKUP FAIL: only ${AVAIL}MB free"
  exit 1
fi

# 3. Copy, and capture the real exit status
rsync -a --delete /home/sean/agents/ "$DEST/agents/$STAMP/" || FAIL=1
sqlite3 /home/sean/agents/state.db ".backup $DEST/db/$STAMP.db" || FAIL=1

# 4. Verify the DB is a readable SQLite file with plausible content
if ! sqlite3 "$DEST/db/$STAMP.db" "PRAGMA integrity_check;" | grep -q "^ok$"; then
  notify "BACKUP FAIL: integrity check on $STAMP.db"
  exit 1
fi

# 5. Only NOW claim success — and post the proof
if [ "$FAIL" -eq 0 ]; then
  SIZE=$(du -sh "$DEST/agents/$STAMP" | cut -f1)
  CHECKSUM=$(sha256sum "$DEST/db/$STAMP.db" | cut -c1-12)
  notify "backup OK $STAMP size=$SIZE db_sha=$CHECKSUM"
else
  notify "BACKUP PARTIAL FAIL on $STAMP"
  exit 1
fi

Key changes:

set -euo pipefail — the script dies on the first real error instead of strolling to the success line.
mountpoint -q — explicitly proves the destination is mounted storage. This one check would have caught all 47 days of failure.
Integrity check on the copied database — PRAGMA integrity_check catches truncated or corrupt copies, not just missing ones.
A positive heartbeat, not just negative alerts. Every successful run pushes a message (I use a tiny webhook to my phone) containing the date, size, and a checksum prefix. If I don't see that message for 36 hours, something is wrong even if no error ever fired. Absence of bad news is not news; I made the backup generate actual news.

The part I skipped the first time: restore drills

Here's the second failure, and it's more embarrassing than the first.

Two weeks after rebuilding the script, I decided to test a restore. The backup was fresh, verified, checksummed, beautiful. And the restore failed — because the backup included the running SQLite database files copied with plain rsync in an older snapshot, and more importantly, my agents' config referenced absolute paths and an environment file that lived outside the backed-up directory. I had a complete copy of data and no copy of the thing that makes the data usable.

The fix was a RESTORE.md in the repo — a literal runbook:

Flash fresh Raspberry Pi OS Lite
apt install the pinned package list (I now generate packages.txt nightly into the backup)
Copy .env from the encrypted secrets folder (which itself was in neither backup — it's now backed up separately with age encryption)
Restore the DB from the newest integrity_check-passing snapshot
Run systemctl start on the units, verify heartbeats within 10 minutes

Then I did the thing I should have done from day one: I ran the drill. Booted a spare Pi from scratch, followed my own document, hit two outdated steps, fixed them, and got to a fully verified restore in 22 minutes. Now the drill runs the first Sunday of every month, and the timer is in cron like everything else.

A backup's real metric isn't "did the copy succeed." It's time-to-restored-service. Mine went from "unknown/infinite" to a tested 22 minutes.

What I'd tell myself six months ago

Silence is a failure mode. Any safety system that only speaks when asked is a system you'll forget to ask. Make it send a heartbeat with content — size, checksum, timestamp — so a missing or malformed heartbeat is detectable.
Verify the destination before writing to it. One mountpoint -q line would have saved 47 days of false confidence.
Untested restores don't exist. The restore drill found two gaps the backup script structurally could not find. Run the drill before you need it, not after.
The unmonitored thing will be the thing that kills you. I had dashboards for agents, alerts for API spend, and heartbeats for cron jobs. The backup ran outside all of it, because it "just worked." That phrase should be a lint error in every ops setup.
Back up the environment, not just the data. DBs and files are half a system. Pinned package lists, env/secrets (encrypted), and a written restore runbook are the other half.

Total cost of the rebuild: one $12 USB drive kept in a different room, about 90 minutes of work, and one monthly 20-minute drill. The SD card that died cost me an evening — instead of the business.

The deeper lesson, and the one that applies way beyond Raspberry Pis: every automation you run — agents included — needs the same treatment. Prove it ran, prove the output is valid, and rehearse the recovery while nothing is on fire.

I write up the specific playbooks in The Solo Operator's AI Agent Playbook — code LAUNCH90 at checkout makes it $1.90. If it doesn't save you 5 hours in week one, reply to the receipt for a refund.

My AI Agent Tried to Install a Typosquatted Package at 3 AM. It Was One Dependency Conflict Away From Owning My Keys.

ULNIT — Mon, 14 Sep 2026 01:03:05 +0000

My agent runs on a Raspberry Pi 4, and for the first two months I let it do whatever it needed to do — including installing its own Python dependencies. It was great. It unblocked itself constantly. I'd wake up to notes like "installed feedparser to handle the RSS task, working now."

Then one Tuesday I looked at an install log and saw a package name I didn't recognize. The agent had tried to install reqeusts — a typosquat of requests — because it had hallucinated the spelling from a scraped Stack Overflow answer, and something on PyPI was waiting for exactly that mistake.

The install failed on a dependency conflict. Pure luck. If it had succeeded, whatever was in that package would have run, with my agent's permissions, on a box that holds API keys for my payment processor, my email, and my DNS provider.

This post is what I changed afterward: how to let an AI agent install software without handing strangers a shell on your infrastructure.

The core problem: agents are gullible installers

A human developer typo-squats rarely because they've installed requests five hundred times and their fingers know the spelling. An agent doesn't have fingers. It reconstructs package names from context — scraped docs, training data, a README it read three tool calls ago. Every reconstruction is a chance to produce reqeusts, python-dotenv, colorsama, or any of the other names that attackers register and seed with malware.

And the agent has three strikes against it that a human doesn't:

It never sees the install page. A human glancing at PyPI notices "last release 2019, 4 downloads, one maintainer." The agent just runs the command.
It installs at 3 AM. There's no second pair of eyes, ever.
It has a reason to say yes. The agent's job is to complete the task. "Install the package" is the path of least resistance to "task complete." Refusing to install something requires a rule it was explicitly given.

What I built instead: a package broker

I didn't ban installs — banned things get worked around, and I'd rather have the agent ask than get creative. Instead every install goes through a small broker script. The agent calls pkg_install(name) as a tool; it never touches pip directly. The pip binary on the box is wrapped so direct invocation fails with a message telling the agent to use the tool.

The broker does five checks before anything touches disk:

1. Exact-name allowlist. I maintain a flat file of ~40 packages my workflows legitimately need. Anything on the list installs immediately, no friction. The list covers 95% of real requests.

2. Fuzzy-match the rejection. If the requested name isn't allowed but is within edit distance 2 of an allowed package, the broker doesn't just say no — it says "rejected: reqeusts is not on the allowlist and looks like a typo of requests. Did you mean requests?" This single change killed almost all hallucinated names. The agent corrects itself and retries with the right spelling.

3. Age and popularity gate for new packages. If the name is genuinely new (not a near-miss of anything allowed), the broker queries the PyPI JSON API and refuses anything where: first release < 90 days ago, or the project has no homepage/repository URL, or the maintainer has exactly one package that appeared this week. Typosquats almost always trip at least one of these.

4. Human approval queue for the rest. Anything that passes the automated gates but isn't already allowed goes into a queue I check once a day from my phone. Approval adds the package to the allowlist permanently, so I only ever evaluate a given package once. Average wait for the agent: a few hours. I've never had a task where that mattered.

5. Full audit log. Every request — allowed, auto-rejected, or queued — is appended to a JSONL file with the agent's stated reason for wanting the package. More on why this mattered below.

The whole broker is about 120 lines of Python. The checks aren't clever; the point is that they're between the agent and the network, and the agent can't route around them.

Sandboxing the install itself

Even an allowed package runs arbitrary code at install time (setup scripts) and at import time. For my setup:

The agent's whole runtime lives in a dedicated user account with no sudo, and its tool sandbox runs as that user.
API keys are not in its environment. They live behind a tiny local proxy that attaches credentials to outbound requests the agent is allowed to make. A malicious package inside the agent's process can use the proxy for whitelisted endpoints; it cannot read the raw keys to exfiltrate them elsewhere.
The Pi's outbound traffic goes through a firewall rule that only allows the specific hosts the proxy and the agent need. An installed package that phones home to a random C2 domain fails at the network layer.

That third one is the check I'd add first if you add nothing else. Allowlist egress on any machine where an LLM can execute code. It converts "malware installed" from a catastrophe into an alert.

The failure that taught me the most

Two weeks after the broker went live, I checked the audit log and found eleven requests in one night for a package called pytelegrambotapi-async-helper — not on my allowlist, not a typo of anything on it. The agent's stated reason, logged verbatim each time: "needed to send the daily report; install rejected; retrying with same package in case of transient error."

Two lessons:

Retry-stupidity is real. The agent treated a policy rejection like a network flake. I'd written the broker's rejection messages politely ("this package is not currently allowed") and the agent read politeness as ambiguity. I changed the rejection text to be blunt and final: "POLICY REJECTION — do not retry. Use an allowed package or explain the need to the operator in your report." Retries dropped to zero. If your agent loops on a rejection, your error messages are too nice.

It was trying to solve a problem I'd created. I'd removed its old notification method in a refactor that day and never gave it a replacement. It wasn't misbehaving; it was resourceful with bad tools. Once I added an allowed notify tool, that class of request disappeared entirely. The audit log is what connected the two facts — without the "reason" field I'd have just seen a suspicious package name and wondered what was wrong with my agent.

What I'd do differently from day one

Start with egress filtering, not package rules. Network allowlisting is dumb, robust, and catches problems you didn't think to write rules for.
Log the agent's reason for every sensitive action, not just the action. The action tells you what happened; the reason tells you what the agent was actually trying to do, which is usually something legitimate you can support properly.
Make policy rejections sound like policy. Anything that reads like a soft failure will be retried.
Don't assume hallucination only affects prose. Package names, CLI flags, config keys — anything the model reconstructs from memory is a typo candidate, and typos in prose are embarrassing while typos in pip install are supply-chain incidents.

The agent still installs what it needs. It just does it through a door I control, and in eight months the door has caught three typosquats and zero legitimate packages. That's a good trade.

The full checklist + scripts are in Ship Safe — The Launch-Day Security Kit — code LAUNCH90 at checkout makes it $1.50.

If you run an agent that can execute code, check your install logs today for package names you don't recognize. Finding nothing is the expected result — and the reason to keep the broker in place anyway.

I Started Version-Controlling My AI Agent's Prompts Like Production Code. The First "Improved" Prompt Broke Three Days of Silent Work.

ULNIT — Sun, 13 Sep 2026 01:03:23 +0000

I started version-controlling my AI agent's prompts like production code. The first "improved" prompt broke three days of silent work.

For months my prompts lived wherever they happened to be written: inline strings in Python scripts, a notes app, one in a README I'd forgotten about, and at least two in my shell history. It worked — right up until it didn't, and when it didn't, I had no way to answer the only question that mattered: what changed?

This is the system I built after that, the failure that forced it, and the parts I'd skip if I were starting over.

The failure: a "better" prompt that wasn't

I run a small support-triage agent on a Raspberry Pi. It reads incoming messages, classifies them, and drafts replies for me to approve. One evening I pasted in a prompt I'd found online — longer, more structured, full of role-setting language. It looked professional. I swapped it in, watched a couple of outputs, thought "seems fine," and went to bed.

Three days later I noticed the drafts had a strange quality: they were polite, thorough, and completely useless for urgent issues. The agent had stopped flagging anything as high-priority. Every ticket got the same calm, apologetic, "we'll look into this shortly" treatment. One customer with a genuinely broken payment waited two days for a human to notice.

The new prompt had a line in it something like "remain calm and reassuring at all times; avoid alarming the customer." The model had interpreted that as "never say anything is urgent." A single clause, buried in paragraph six of a prompt I'd copied without diffing against the old one.

Here's what actually stung: I couldn't tell you what the old prompt said. It was gone — overwritten in the script. No history, no diff, no record of what behavior I'd lost. I spent an evening reconstructing it from memory and a stale backup, and the reconstructed version was measurably worse at classification than the original had been. I never fully got it back.

The system: prompts are code, treat them like code

After that I moved every prompt into a git repo with three rules.

Rule 1: One prompt per file, named for the job.

prompts/
  triage-classify.md
  triage-draft-reply.md
  recon-summarize-findings.md
  digest-morning-brief.md

Each file is pure Markdown with a small YAML header:

---
name: triage-draft-reply
model: gpt-4o-mini
version: 7
last_changed: 2026-08-30
owner_task: reply drafting only — never classifies priority
---

That owner_task line matters more than it looks. Half my early prompt bugs came from one prompt quietly doing two jobs. When the drafting prompt started classifying priority as a side effect, nobody had decided that — it just happened.

Rule 2: Never edit a prompt without a fixture run.

For each prompt I keep 5–10 canned inputs with expected properties of the output. Not unit tests with exact string matches — that's brittle nonsense with LLMs. Instead, cheap property checks:

def check_urgent_flag(output):
    # the angry-payment-failed fixture MUST be classified urgent
    assert "urgent" in output.lower() or "high" in output.lower()

def check_low_priority(output):
    assert "urgent" not in output.lower() and "high" not in output.lower()

FIXTURES = [
    ("fixtures/angry_payment.txt", check_urgent_flag),
    ("fixtures/spam_nigerian_prince.txt", check_low_priority),
    ("fixtures/normal_question.txt", check_no_apology_storm),
]

The third fixture exists specifically because of the failure above. I added an assertion that catches apology-storm output — five consecutive hedging phrases — and it has caught two regressions since.

The whole suite costs me about four cents to run. I run it before every prompt change, in a pre-commit hook, so I literally cannot ship a prompt edit without the fixtures passing.

Rule 3: The commit message says what behavior changed, not what text changed.

git diff tells me the text. The commit message tells me why:

v6->v7: stop the model from apologizing before answering.
Added "no preamble, first sentence = the answer".
Fixture: normal_question now passes apology check.

Three months later, when something feels off, git log prompts/triage-draft-reply.md reads like a behavioral changelog. When I broke things in August, I had nothing.

What the system caught that I didn't expect

The fixtures aren't just regression insurance — they surfaced problems that had been quietly costing me for weeks.

Model drift. My provider updated a model version and my classification accuracy on the fixture set dropped from 9/10 to 6/10 overnight. Because I had the fixtures, I knew it was the model, not my prompt, and I pinned the old version within an hour. Without them I'd have spent a day "improving" a prompt that was fine.
A prompt that had quietly doubled in size. Every time I patched an edge case, I added a sentence. Version 4 of the drafting prompt was 700 words of accumulated scar tissue. Reading the diff history made the bloat obvious, and I rewrote it down to 200 words — which fixture-tested better than the long version.
Two prompts that disagreed with each other. My triage prompt told the agent to be concise; my drafting prompt told it to be thorough. The agent was resolving that contradiction differently on different days. Neither prompt was wrong; the pair was. You only see that when all prompts live in one repo, side by side.

What I'd do differently / what I got wrong

Honest part, because this system isn't free.

I over-engineered the first version. My initial setup had a database of prompt versions, a diff UI, and "rollbacks" as first-class objects. It took a weekend to build and I abandoned it in two weeks. Git already does all of it. The version that survived is a folder of Markdown files and a 60-line test script. If you take one thing from this article: the boring version is the version that lasts.

Fixtures rot. Three of my original ten fixtures encoded assumptions that stopped being true (an old pricing tier, a retired feature). Stale fixtures that fail for the wrong reason train you to ignore failures — the exact failure mode the system exists to prevent. I now review the fixture set monthly, same as I'd review tests.

It doesn't fix prompt quality. Version control tells you what changed and whether behavior regressed. It does not make prompts good. For a long time I mistook "I can see the history of this bad prompt" for "this prompt is good." The improvements came from actually studying which phrasings worked, one experiment at a time.

It's overkill below ~5 prompts. If you have one agent with one prompt, a .txt file and discipline is enough. The system pays for itself when prompts multiply and you can no longer hold them all in your head — which, for me, happened around prompt number seven.

The takeaway

The failure that started this wasn't a model failure. It was an ops failure: I changed production behavior with no diff, no history, and no test. Every developer already knows why that's reckless with code. Prompts are the same thing — executable instructions that shape real behavior — they just don't feel like code because they're written in English.

Put them in git. Write five fixtures each. Make the commit message about behavior. That's the whole system, and it takes an afternoon.

Collecting and refining prompts is the part that actually takes time — I keep every prompt that survived fixture-testing in one place, organized by job, along with the notes on why each phrasing works. All 100 prompts are in The Agent Prompt Vault — $3, lifetime updates. Steal the ones that fit your workflow.

I Made My AI Agent Ask Permission Before Spending or Sending Anything. By Day 3 I Was Ready to Remove It.

ULNIT — Sat, 12 Sep 2026 01:03:19 +0000

Last month I wrote about an agent that charged a customer twice in eleven seconds. The fix wasn't a better model or a cleverer prompt — it was a rule I resisted for weeks because it felt like admitting defeat:

Nothing that moves money or leaves the building happens without a human saying yes.

So I built an approval gate. Every action my agents take gets classified: read-only stuff runs free, anything that sends an email, posts publicly, calls a payment API, or touches production config gets parked in a queue until I approve it from my phone.

Day 1 felt like genius. Day 3 I had 47 pending approvals, I was rubber-stamping them without reading, and I realized I had rebuilt the exact problem I was trying to solve — just with extra steps.

Here's what actually worked after I redesigned it, including the part where I nearly gave myself approval fatigue so bad it became worse than no gate at all.

The naive version (don't do this)

My first implementation was dead simple. Before any "risky" tool call, the agent writes a row to a SQLite table:

RISKY_TOOLS = {
    "send_email", "post_to_social", "create_refund",
    "charge_card", "deploy", "write_production_db",
}

def request_approval(agent_id: str, tool: str, payload: dict) -> str:
    ticket_id = str(uuid.uuid4())[:8]
    db.execute(
        "INSERT INTO approvals (id, agent, tool, payload, status, created_at) "
        "VALUES (?, ?, ?, ?, 'pending', datetime('now'))",
        (ticket_id, agent_id, tool, json.dumps(payload)),
    )
    notify_me(ticket_id, agent_id, tool, payload)  # push notification
    return ticket_id

The agent loop then polls for a decision:

ticket = request_approval(agent_id, tool_name, args)
decision = wait_for_decision(ticket, timeout=3600)  # 1 hour
if decision != "approved":
    return {"status": "blocked", "reason": f"action required approval: {ticket}"}

And a 30-line Flask endpoint on my Pi lets me approve or deny from my phone. That's the whole system. It works. It is also, as configured above, a machine for generating 40+ notifications a day, because I had classified actions by tool instead of by consequence.

Why day 3 almost broke it

Three things went wrong, and all three were my fault:

I gated reads that looked like writes. My support agent's send_email tool was used for both customer replies and an internal daily digest. Every digest needed approval. Pointless.
Low-stakes actions drowned high-stakes ones. A $3 refund request and a "post this to X" request arrived in the same channel with the same urgency. After two days I stopped reading the payloads. I was approving on muscle memory — which is exactly the failure mode an approval gate exists to prevent. A gate you don't read is theater.
No timeout policy meant silent stalls. When I forgot my phone, agents sat blocked for hours. Customers got replies at 11 PM. The gate had made the product worse, not safer.

The honest lesson: an approval system is a UX problem, not a security problem. If approving is annoying, you will degrade your own controls within a week. I've watched myself do it in real time.

The redesign: tier by consequence, not by tool

I now classify every action into three tiers, and only one of them involves me.

Tier 1 — Reversible and internal. Run free. Reading data, drafting content to a staging area, internal DB writes that have an undo path. No notification, no wait.

Tier 2 — Reversible but external. Auto-approve with a delay and an audit log. Example: routine support replies. The agent sends, but the message goes out through a queue with a 10-minute delay, and a second cheap LLM pass flags anything that mentions refunds, pricing, or legal language. Flagged items get promoted to Tier 3. In practice ~4% of messages get flagged, and the flagger has caught real problems twice — once when an agent promised a "lifetime" discount it had no authority to offer.

Tier 3 — Irreversible or money-moving. Hard block until human approval. Charges, refunds above $5, public posts, deploys, production schema changes, anything touching credentials. These get a push notification with a diff-style summary: what will happen, to whom, for how much, and why the agent thinks it should.

The tiering logic is just a function — no framework needed:

def tier_for(tool: str, payload: dict) -> int:
    if tool in ("charge_card", "deploy", "write_production_db"):
        return 3
    if tool == "create_refund":
        return 3 if payload.get("amount_cents", 0) > 500 else 2
    if tool == "send_email":
        return 2  # queue + flagger may promote to 3
    if tool in ("post_to_social",):
        return 3
    return 1

The numbers matter less than the principle: tier by blast radius and reversibility, not by which function got called.

Three details that made it survivable

Batched approvals. Tier 3 requests now arrive in a twice-daily digest unless they're flagged urgent (a live customer waiting, a deploy window). My approval count went from ~47/day to ~6/day, and — this is the part that surprised me — I actually read them again. Friction per item went up; total friction went down.

Expiry with a safe default. Every pending approval expires after 4 hours and defaults to deny, and the agent is required to tell the customer "this is taking longer than usual, I'll follow up" rather than going silent. A stale approval queue is a lying queue.

The agent must argue its case. The approval payload includes the agent's reasoning and the specific evidence it used (order ID, email thread, exact amount). Writing that field forced me to make the agent's context inspectable — which paid off separately, because it's how I caught a hallucinated order number in week two. If the agent can't cite evidence for an irreversible action, that's a deny on its own.

What it costs, honestly

Latency. Tier 3 actions can wait hours, and if you're building something customer-facing you need the "I'll follow up" paths to be real. I've lost the odd sale to a slow approval. I'll take that over the alternative — the double-charge incident cost me more in trust (and one very reasonable but very public complaint) than every delayed approval combined.

Maintenance. The tier function is policy, and policy drifts. Once a month I read the audit log end-to-end and ask two questions: did anything Tier 1/2 do damage? Did anything Tier 3 get denied that shouldn't have been? Both answers change the tiers. It's maybe an hour a month and it's the highest-leverage hour in my whole agent stack.

If you build one thing from this post

Don't gate tools. Gate consequences. Write down every action your agent can take, score each one "can I undo this in under five minutes?" and "does this touch money, reputation, or credentials?" — and put a human in the loop only where both answers are bad. Then design the approval UX as if your future tired self is the user, because your future tired self is definitely the user, and a tired approver is an auto-approver.

The gate is not the point. Reading the gate is the point.

I write up the specific playbooks in The Solo Operator's AI Agent Playbook — code LAUNCH90 at checkout makes it $1.90. If it doesn't save you 5 hours in week one, reply to the receipt for a refund.

I Pretended to Attack My Own Infrastructure for a Saturday. I Found 6 Doors I Forgot Were Open.

ULNIT — Fri, 11 Sep 2026 01:03:13 +0000

I've been shipping fast for the past year — solo, no ops team, no security engineer. Last month I decided to spend one Saturday pretending to be an attacker targeting my own infrastructure. Not a formal pentest. Just the lazy-but-methodical recon any bored script kiddie does in their first ten minutes: certificate transparency logs, a port scan, some curl requests, and a lot of guessing subdomain names.

I expected to find one or two embarrassing things. I found six. Three of them were doors I didn't even remember leaving open.

This post is the full list, the exact commands I used, and what I changed afterward. None of this requires paid tooling. All of it took under four hours.

Why bother? Because attackers don't find your bugs, they find your leftovers

The things that got me were never the parts of the system I thought about carefully. My payment flow is hardened — I spent real time on it. What got me were the artifacts of past shipping: a staging environment from a launch six months ago, a debug endpoint added during one bad Tuesday, a subdomain pointing at a service I'd deleted.

Attack surface isn't what you build. It's what you forgot you built.

The recon loop I ran (steal it)

1. Ask Certificate Transparency what you own

Every TLS cert you've ever issued is publicly logged. This is the single highest-signal, zero-effort step:

curl -s "https://crt.sh/?q=%25.yourdomain.com&output=json" \
  | jq -r '.[].name_value' | sort -u

This gave me 23 subdomains. I honestly believed I had about 8.

2. Find out which ones still resolve

for sub in $(cat subs.txt); do
  ip=$(dig +short "$sub" | head -n1)
  [ -n "$ip" ] && echo "$sub -> $ip"
done

Dead subdomains pointing at deprovisioned services are their own risk class (subdomain takeover), but the live ones are where the surprises hide.

3. Port-scan the IPs you just discovered

nmap -sV --top-ports 1000 <your-ip>

4. `curl` everything with and without trailing paths

curl -sI https://each-subdomain.yourdomain.com/
curl -s https://each-subdomain.yourdomain.com/robots.txt

robots.txt is a map of what you asked crawlers to avoid — which is a decent proxy for what you didn't want seen.

That's the whole loop. crt.sh → dig → nmap → curl. Free, thirty minutes, and it found everything below.

What I found

1. A staging subdomain, live, six months stale. staging.mydomain.com still pointed at an old VPS, still had HTTP basic auth — with the same password as one of my dev accounts I'd reused in a hurry. The code on it predated three security fixes. Gone within the hour. The lesson isn't "staging is bad," it's: staging environments need an expiry date written down the day you create them.

2. A .git directory served over HTTP. On that same stale box, /.git/config returned 200. That means anyone could have reconstructed the full repo history — including a .env committed in month one and later removed. Removed from git, but git history is forever unless you rewrite it. I'd done the "delete the file and commit" dance and told myself the key was gone. It wasn't. (It had been rotated by then, by luck not by process — more on that in the failure section.)

3. A debug endpoint on production. /api/debug/state dumped internal queue lengths, worker hostnames, and the app version. Harmless-looking, but it told an attacker exactly which CVEs to look up and exactly how my workers were named. It was added during one bad incident, gated behind nothing, and forgotten.

4. A Raspberry Pi running a dashboard, port-forwarded, no auth. This one hurt because it was my homelab Pi — I'd port-forwarded 8080 during setup to test remote access, then "temporarily" left it for eleven months. The dashboard exposed job names, last-run outputs, and (indirectly) which internal services exist. My own logs told a stranger more about my architecture than my public site did.

5. Verbose error pages leaking stack traces. One route threw a 500 with a full Python traceback, including file paths and a partial connection string. Fixed by setting a generic error handler I should have had on day one.

6. An SSH port open on the VPS with password auth enabled. Key-based auth was set up and in use — but PasswordAuthentication yes was still the default. Brute-force attempts in auth.log were constant; I'd just never looked. One-line fix, eleven months overdue.

The honest failure section

Here's the part that stings.

Finding #2 — the exposed .git — should have been caught by me in month one. I knew I'd committed a .env early on. I knew I'd "cleaned it up" with a follow-up commit. I even remember thinking, briefly, I should check whether that's actually safe. And then a customer email came in and I moved on.

The reason it didn't become a catastrophe is not that I was careful. The key in that .env had been rotated four months earlier because a different incident forced me to. If that rotation hadn't happened, my recon Saturday would have been a breach-postmortem Saturday instead.

That's the uncomfortable lesson: my security posture was being maintained by accidents and adrenaline, not by process. Things got fixed when they broke loudly. Nothing got fixed quietly — which is exactly the category all six findings fell into. They were all silent. None of them would ever have thrown an error, sent an alert, or shown up in my analytics.

So the real change I made wasn't any individual fix. It was putting the recon loop on a calendar: the first Saturday of every month, the crt.sh → dig → nmap → curl sequence, thirty minutes, written into the same cron-like schedule as my backups. Findings go in a list. The list gets reviewed before any new launch, because launches are when new leftovers get created.

Two smaller habits came out of it:

Every "temporary" thing gets a deletion date in its commit message or DNS record description. TEMP-until-2026-04 staging box shows up in crt.sh output like a confession.
Anything bound to a port gets asked one question: what does an unauthenticated stranger see? If the answer isn't "nothing" or "a login page," it doesn't ship.

What I'd tell you if you're solo and shipping fast

You are not going to build a perfect perimeter. You don't have time, and honestly you don't need one — you need to not be the easiest target in the attacker's tab list. Almost every finding above was a leftover, not a design flaw. Leftovers are cheap to find (crt.sh is free) and cheap to kill (most of mine were rm -rf or a config one-liner).

The gap between "I think my stuff is fine" and "I know what my stuff looks like from outside" is about thirty minutes of commands you can run today. Close that gap before someone else closes it for you.

The full checklist + scripts are in Ship Safe — The Launch-Day Security Kit — code LAUNCH90 at checkout makes it $1.50.

If you run the loop above and find something scary, that's the system working. The scary part is the month where you find nothing and you're not sure whether it's clean or whether you scanned the wrong domain.

I Gave My AI Agent Long-Term Memory. It Started Confidently Lying to Customers About Facts From March.

ULNIT — Thu, 10 Sep 2026 01:03:39 +0000

I gave my AI agent long-term memory in an afternoon. It took me two weeks to realize memory was the problem, not the solution.

The pitch was irresistible. My support agent forgot everything between conversations — a customer would explain their setup on Monday, and by Wednesday the agent was asking them the same three questions again. So I did what every builder does: I bolted on a memory layer. Embed the conversation summary, store it in a vector DB, retrieve the top 5 relevant memories before every reply. Classic RAG-over-history. It worked beautifully in my tests.

Then it started confidently telling customers things that used to be true.

The incident that made me look

A customer on the annual plan emailed asking to change their billing email. My agent replied — warmly, fluently, completely wrong — that they were on the monthly plan and could simply cancel anytime. The customer, understandably, escalated: "Are you telling me I don't have an annual contract?"

I dug through the logs. Here's what happened:

In March, the customer had been on the monthly plan. The agent summarized that conversation and stored: Customer is on monthly plan, considering upgrading.
In May, they upgraded to annual. That happened in the billing system, not in a conversation — so no memory was ever written.
In September, the memory retrieval surfaced the March fact. It was semantically relevant (billing + plan), highly similar to the question, and ranked #1 of 5.
The model did what models do: it trusted its context. The retrieved memory outranked the vague instructions in my system prompt about "checking current plan details via the account tool."

The memory wasn't wrong when it was stored. It was stale. And my architecture had no concept of staleness whatsoever. A fact from March and a fact from five minutes ago had exactly the same authority in the prompt.

This is the failure mode nobody warns you about with agent memory: you're not building a memory, you're building a slowly-rotting cache with no TTL and no invalidation. Everyone who has done backend work knows a cache without invalidation is a bug factory. Somehow we collectively forgot that the moment we started calling it "memory."

The honest part: I'd seen the warning signs and ignored them

For a full week before the incident, small things were off. The agent twice referred to a pricing page that I had redesigned in July — describing buttons and plans that no longer existed. Once it apologized for an outage from May as if it were recent ("sorry about the disruption earlier this week").

I read those transcripts and thought hallucination, need a better model. I even bumped up to a more expensive tier for a few days, which cost me money and fixed nothing, because the model wasn't hallucinating. It was accurately reporting what its context contained. The context was the liar.

I lost roughly a week to the wrong diagnosis because "the memory system is feeding it dead facts" wasn't a hypothesis I considered. When you build a feature, you don't naturally suspect the feature. Lesson: when an agent says something confidently wrong, the first question shouldn't be "why did the model fail" — it should be "what was in the prompt, and who put it there?" Log the full assembled context, not just the user message and the reply. I now store the retrieved memories alongside every conversation, which is what let me root-cause this in ten minutes instead of ten days.

What I changed

Five fixes, in the order I'd recommend them:

1. Every memory gets a timestamp and a type. I split memories into two kinds: observations ("customer mentioned they use a Raspberry Pi at home") and state ("customer is on the monthly plan"). Observations age gracefully. State doesn't — state describes something that can change outside the conversation.

2. State memories expire. Anything classified as state gets a TTL: 7 days for plan/account facts, 30 days for softer things like project status. When retrieval pulls an expired memory, it's either dropped or re-fetched from the source of truth. Yes, this means classifying every memory at write time — one extra LLM call, about half a cent. Cheap compared to one furious annual-plan customer.

3. Tools beat memories. The real fix for the incident: plan, billing status, and subscription details are now never answered from memory. The system prompt says: "If the question involves current account state, call get_account — retrieved memories about account state are for context only and MUST NOT be stated as fact." Retrieval can inform tone ("this customer has been with us a while"); the API informs facts.

4. Memories are written as snapshots, not assertions. Instead of Customer is on monthly plan, the agent now stores As of 2026-03-14, customer said they were on the monthly plan. Sounds pedantic. It's not: when that string lands in a prompt, the model handles it completely differently. It hedges, it verifies, it calls the tool. Phrasing the memory with its date attached was the single highest-leverage change I made, and it cost me an afternoon of rewriting the summarization prompt.

5. A nightly reaper. A cron job on the Pi reviews memories older than 90 days and deletes any that were never retrieved. Turns out most memories are write-only. If retrieval never surfaced a memory in three months, it's not a long-tail gem — it's noise with an embedding attached.

The bigger lesson

Agent memory has the exact same failure modes as every caching system that ever existed: stale reads, no invalidation, write amplification, and trust boundary confusion (cache treated as source of truth). The industry gave it a friendlier name so we'd stop applying thirty years of hard-won distributed-systems intuition to it.

If you're adding memory to an agent, ask yourself the cache questions before the AI questions:

What's the invalidation policy?
What happens on a stale read?
Which fields must never be served from cache?
Can I tell, after the fact, which cached values influenced a given response?

If you can't answer those, you don't have a memory system yet. You have a time-delayed hallucination engine.

After the five fixes, the agent hasn't stated a stale fact as truth in six weeks — and the two times retrieval surfaced something expired, the logs show it called get_account instead. That's the behavior you want: memory for rapport, tools for truth.

I write up the specific playbooks in The Solo Operator's AI Agent Playbook — code LAUNCH90 at checkout makes it $1.90. If it doesn't save you 5 hours in week one, reply to the receipt for a refund.

My AI Agent Kept Calling the Wrong Tool. I Blamed the Model for a Week — the Bug Was in 6 Lines of JSON

ULNIT — Wed, 09 Sep 2026 01:03:57 +0000

For seven days I was convinced I'd picked the wrong model.

My support agent — the one that looks up orders, checks refund eligibility, and escalates to me when it's out of its depth — kept doing something maddening: when a customer asked "where's my order?", it would call search_knowledge_base instead of get_order_status. It would return a polite, fluent, completely useless answer about shipping policies in general. The customer's actual order? Never looked up.

I did what any reasonable solo operator does: I blamed the model. I upgraded to a bigger one. I cranked the temperature down. I rewrote the system prompt three times. I added few-shot examples of correct behavior. The failure rate dropped a little, then plateaued at roughly 1 in 5 conversations going sideways.

The actual fix took about twenty minutes, cost $0, and lived nowhere near the system prompt. It lived in the tool definitions — six lines of JSON I'd written in a hurry and never revisited.

Here's everything that week taught me about the part of prompt engineering nobody talks about.

The part of the prompt you forgot you wrote

When you give an agent tools, you're not just giving it capabilities. You're giving it a menu with descriptions, and the model picks from that menu on every single turn. The tool schema — name, description, parameter definitions — is prompt text. It carries the same weight as your system prompt. Most of us treat it like API documentation we fire off and forget.

My original search_knowledge_base tool looked something like this:

{
  "name": "search_knowledge_base",
  "description": "Searches for information to help answer the customer's question. Use this tool whenever you need information you don't have.",
  "parameters": { "query": { "type": "string" } }
}

Read that description again, from the model's point of view, on the turn where a customer asks "where's my order?"

"Use this tool whenever you need information you don't have."

The model doesn't have the order status. It needs information it doesn't have. By my own instructions, this was the correct tool. I hadn't written a description — I'd written a catch-all magnet that hoovered up every ambiguous turn.

Meanwhile get_order_status was described as: "Gets order status." Three words competing against a sentence that promised to solve everything. Of course the flashy generalist beat the terse specialist. Models are not reading your tools as an API reference; they're reading them as competing sales pitches.

What I changed

Once I understood that tool descriptions are prompts, the rewrite was mechanical. Four rules, all earned the hard way:

1. Describe when NOT to use the tool. Negative space does more work than positive. search_knowledge_base became:

"description": "Searches general policy and product documentation (shipping times, warranty rules, how-tos). ONLY for general questions. NEVER use for anything about a specific customer's order, account, or refund — use get_order_status or get_refund_eligibility for those, even if the customer hasn't given an order number yet."

That "even if the customer hasn't given an order number yet" clause fixed more failures than everything else combined, because it named the exact ambiguity the model was resolving wrong.

2. Let tools declare their prerequisites. get_order_status became: "Looks up a SPECIFIC customer's order: current status, tracking link, and delivery estimate. ALWAYS use this first when the customer asks about their own order. If you don't have an order number or email, ask the customer for one — do not fall back to searching the knowledge base." Now the model knows the tool exists precisely for the question it's staring at, and knows what to do when an argument is missing.

3. Rename tools so the name does half the work. I had escalate and escalate_to_human in two versions of the agent. The model conflated escalate with "handle this seriously" and sometimes called it for routine questions. Renaming it transfer_to_human_agent_and_end_conversation made misfires nearly vanish — the name now spells out the consequence. Names are read every turn; make them self-documenting.

4. Constrain parameters with enums and formats, not prose. My priority parameter used to be "priority: string". The model invented values like "high-ish" and "URGENT!!". Now it's {"enum": ["low", "normal", "high"]} and downstream code stopped crashing on Tuesdays. Wherever you're parsing a tool argument yourself, an enum or a stated format ("ISO 8601 date, e.g. 2026-09-01") is cheaper than any amount of retry logic.

The failure that made me log tool calls in the first place

Honest part of the story: I only found any of this because of an earlier, worse failure.

For the first two weeks, the agent ran with no structured logging — just the raw conversation transcript. When a customer complained the bot was "useless," I'd read the transcript, see a fluent and plausible answer, shrug, and blame the customer's phrasing. It took a support thread where the customer pasted their side of the conversation — ten messages, order number given in message two, never used — for me to realize I was debugging blind.

So I added one log line per tool call: timestamp, tool name, arguments, and a one-line summary of the preceding customer message. Twenty lines of Python. The pattern jumped out within an hour: every wrong-tool call happened when the customer's message mentioned both a specific order and a general policy word ("when do refunds usually post? I ordered #4471"). The knowledge base tool's description contained the word "information" and the model pattern-matched to it.

If you take one thing from this post: log tool calls, not just transcripts. The transcript tells you the agent was wrong. The tool log tells you why, and "why" is the only thing you can fix.

What I'd tell myself a week ago

Your system prompt is maybe 40% of the prompt your model actually sees. Tool schemas, tool results, and error messages from your own code are the rest. Audit all of it.
Every tool description competes with every other tool description. Write them as a set, not one at a time. If two tools could plausibly answer the same query, say explicitly which one wins and why.
Vague descriptions don't just fail quietly — they succeed loudly, with fluent wrong answers that look fine in a transcript.
The model is not stupid. It's obedient. It did exactly what six lines of JSON told it to do.

The failure rate went from ~20% of conversations to under 2% on the day I shipped the rewrite. No model upgrade, no temperature tuning, no extra cost per call.

Tool descriptions are prompts, and prompts are worth versioning, testing, and stealing good ones from. All 100 prompts are in The Agent Prompt Vault — $3, lifetime updates. Steal the ones that fit your workflow.

I Launched on Product Hunt. Four Hours Later, Someone Was Replaying My Checkout Webhook.

ULNIT — Tue, 08 Sep 2026 01:03:28 +0000

The launch itself went fine. Traffic spike, a few hundred signups, dopamine. Then at around hour four, my monitoring pinged me with something that made my stomach drop: the same webhook event, delivered to my server 61 times, from 9 different IPs.

It was a checkout.completed event from my payment processor. Signed. Valid. And completely fake.

Whoever sent it had captured one real webhook from somewhere — maybe their own purchase, maybe a leaked example — and was replaying it at my endpoint, hoping my server would just... believe it. And the worst part? For the first two days of my launch, it would have. I'd shipped the signature verification and nothing else. No timestamp check, no idempotency, no rate limit on the webhook route.

The attacker's version of the replay was slightly malformed (they'd re-signed it with a guessed secret, so HMAC verification actually rejected it — my one piece of luck). But it forced me to spend that evening auditing every unauthenticated door into my system. Here's what I found, what I fixed, and the exact patterns I now refuse to launch without.

Why webhooks are the softest target on launch day

Launch day is the one day your attack surface gets crowdsourced attention. Thousands of people hit your site, and a few of them are curious in ways you don't want. Authenticated routes are mostly fine — they're behind your login. But webhooks are, by design, endpoints anyone on the internet can POST to. Your payment processor needs to reach them without logging in.

That makes them the classic "door that's unlocked because the delivery guy needs to get in." And most solo founders (me, pre-launch) treat them like internal plumbing instead of what they are: a public API that can credit accounts, mark orders paid, or trigger fulfillment.

If an attacker can get your server to accept a fake payment.completed, they get your product for free. If they can replay a real one, they might get it multiple times — duplicate credits, duplicate provisioning, duplicate emails to a customer who now thinks they bought three subscriptions.

The four checks every webhook endpoint needs

1. Verify the signature (table stakes — and do it in constant time)

Every serious payment processor signs webhooks with an HMAC of the raw body. You must verify against the raw request body, not a parsed-and-re-serialized JSON object — key ordering and whitespace differences will make valid signatures fail and, worse, tempt you to "fix" it by skipping verification.

import hmac, hashlib

def verify_signature(raw_body: bytes, signature: str, secret: bytes) -> bool:
    expected = hmac.new(secret, raw_body, hashlib.sha256).hexdigest()
    # constant-time compare — never use == on signatures
    return hmac.compare_digest(expected, signature)

Two mistakes I've seen (and made) here: comparing signatures with == (timing attack, low risk but free to fix), and verifying a JSON round-trip instead of the raw bytes (breaks randomly, gets "fixed" by deleting the check).

2. Reject stale timestamps (this kills replay)

A valid signature proves the sender knows your secret — it says nothing about when the event was created. Replay is only possible because old events stay valid forever. Almost every processor includes a timestamp in the signed payload or header. Check it:

import time

def reject_stale(timestamp: int, tolerance_seconds: int = 300) -> bool:
    return abs(time.time() - timestamp) <= tolerance_seconds

Five minutes of tolerance is plenty. Now a captured webhook is only useful for 300 seconds, which turns "replay forever" into "race the clock" — and attackers don't like races they usually lose.

3. Store event IDs and make processing idempotent

Even with 1 and 2, legitimate processors will deliver the same event more than once — retries after timeouts are normal. Your handler must be idempotent:

def handle_webhook(event: dict) -> None:
    event_id = event["id"]
    if redis.set(f"webhook:{event_id}", "1", nx=True, ex=86400 * 7):
        process_event(event)  # first time only
    # else: already processed, return 200 anyway

The SET ... NX pattern is atomic — no race between check and insert, even with multiple workers. Note the subtlety: you return 200 for duplicates, not an error. Error responses trigger more retries, and a retry storm against a handler that errors is how you DDoS yourself on launch day.

This one also saved me from a bug I'd already shipped: my fulfillment code credited the user's account inside the webhook handler. Before idempotency keys, a single retried delivery would have double-credited. I found it in my logs before an attacker did — 48 hours after launch, a processor timeout caused a retry, and one early customer got two months of access for the price of one. I caught it because the numbers didn't reconcile, refunded myself the embarrassment, and shipped the Redis guard the same night.

4. Rate-limit the route like it's public — because it is

Per-IP rate limiting on the webhook endpoint costs ten lines and turns "61 requests from 9 IPs in an hour" into "9 IPs politely told to go away." Something like:

from flask_limiter import Limiter
limiter = Limiter(app=app, default_limits=[])

@app.route("/webhooks/stripe", methods=["POST"])
@limiter.limit("30/minute", methods=["POST"])
def webhook():
    ...

Legitimate processors deliver in bursts, not floods — 30/min per IP is generous. Scanners and replayers hit that ceiling instantly and show up in your rate-limit logs, which doubles as a free early-warning system.

The honest failure section: what I got wrong first

I want to be straight about the sequence, because the "four checks" list above makes me sound more competent than I was.

Version 1 (pre-launch): no signature check at all. I told myself I'd "add it before real traffic." Classic.

Version 2 (launch morning): signature check added at 6 AM, raw-body bug included — it rejected legitimate webhooks about 30% of the time because I was verifying re-serialized JSON. My first three paying customers' events bounced. The processor retried, and luck (retry + eventual success) covered for a bug that was silently losing revenue events.

Version 3 (that evening, post-replay-attack): raw body fixed, timestamps checked, idempotency keys in. The double-credit incident above happened after this — because I'd assumed my processor "probably never retries." It does. Assumptions about third-party behavior are bugs with a delivery date.

The pattern in all three: I treated the webhook as plumbing until the internet treated it as a target. The fixes weren't hard — none of this took more than a few hours total. The cost was entirely in doing them reactively, with an attacker's timing instead of mine.

The 15-minute pre-launch webhook drill

If you're launching something with payments soon, here's the compressed version of what I learned the slow way:

curl -X POST your own webhook endpoint with a garbage body. If it returns anything other than 4xx, stop and fix that first.
Grab a real signed payload from your processor's dashboard/test mode. Replay it twice from your terminal. The second one must be a no-op (same 200, no duplicate side effects).
Replay it with the timestamp edited. Must be rejected.
Hammer it with 60 requests in a minute from one IP. Rate limiter must fire.
Check your logs: can you distinguish "duplicate rejected" from "invalid signature" from "rate limited"? If they all look the same, your future incident response will be guesswork.

Five tests, fifteen minutes, and it covers the exact attack I got hit with — plus the two bugs I hit myself.

The full checklist + scripts are in Ship Safe — The Launch-Day Security Kit — code LAUNCH90 at checkout makes it $1.50.

Launch day should be stressful because of traffic, not because of trespassers.

I A/B Tested My AI Agent's System Prompt Against an 11-Minute Version. The Sloppy One Won.

ULNIT — Sun, 06 Sep 2026 01:03:12 +0000

For two weeks I ran the same AI agent — same model, same tools, same customers — with two different system prompts. Half the conversations ran on the prompt I'd spent a weekend polishing. The other half ran on a prompt I'd written in eleven minutes and was mildly embarrassed by.

The embarrassing one won. Not slightly. It won on every metric I cared about: fewer escalations to me, fewer hallucinated commitments, and — the one that actually stung — customers rated the conversations more helpful, even though the polished prompt was objectively "better written."

Here's what the experiment looked like, what the data said, and the four prompt changes that actually moved the needle.

The setup (boring but necessary)

I run a small support/research agent for my one-person business. It answers customer emails, drafts replies I approve, and handles routine questions about my products without me.

Before the experiment, the agent used a prompt I'd iterated on for weeks. It had role framing ("You are a senior customer success engineer"), tone guidelines, a detailed persona, examples of good replies, even a little motivational line about delighting customers. It read like a job description written by someone who'd read a lot of LinkedIn. ~1,400 tokens of careful prose.

I'd been quietly unhappy with it. Escalation rate hovered around 18% — nearly one in five conversations got punted to me with "I'm not confident enough to answer this." Reply drafts were polite but vague. Occasionally the agent would promise things that weren't true (a shipping date, a feature) with the serene confidence of a press release.

So I wrote a challenger prompt. I gave myself fifteen minutes and one rule: no persona, no adjectives, only constraints and procedures. It came out looking like a laminated card a dispatcher keeps next to a phone:

You answer customer emails about [product].

RULES:
1. If the answer is not in KNOWLEDGE below, say you'll follow up. Do not guess. Never invent dates, prices, or features.
2. Prices: only the ones listed in KNOWLEDGE. If asked about anything else, escalate.
3. Refunds: you may offer a refund ONLY if the customer paid within 14 days AND the request is your first reply. Otherwise escalate.
4. Reply length: under 120 words. No exclamation marks. No "I hope this helps."
5. Before sending, check your reply against rules 1-3. If it violates one, fix it, then send.

ESCALATE (write "ESCALATE:" and stop) if:
- customer is angry (any insult, all-caps, or the word "lawyer")
- the email asks you to do something (change data, issue credit, modify account)
- two of your replies failed to resolve the issue

KNOWLEDGE:
[pasted docs, price list, FAQ — the single source of truth]

That was it. ~350 tokens. No identity, no vibe, no motivation. I split incoming traffic 50/50 with a one-line router, logged every conversation to SQLite, and waited fourteen days. 214 conversations total. Enough to see signal, not enough for a journal paper — take the numbers as directional, not scientific.

The results

Metric	Polished prompt	Dispatcher prompt
Escalation rate	18.2%	9.4%
Made-up commitments (manual review)	7 instances	1 instance
Median reply length	214 words	96 words
Customer follow-up needed	41%	29%
"Helpful" rating (1-5, asked in a footer link)	3.6	4.1

The single made-up commitment in the challenger arm: the agent told a customer their issue "would be fixed in the next update." There was no update. Even rules-based prompts leak — I'll get to that in the failure section.

But the headline was the escalation rate halving. The polished prompt's persona — "senior customer success engineer" — apparently came with an implied obligation to have an answer. The agent would rather produce plausible mush than admit it didn't know. The dispatcher prompt had no ego. "Say you'll follow up" was just rule 1.

And customers preferred the shorter, blunter replies. The 120-word cap did more for satisfaction than all my tone guidelines combined. Nobody has ever thanked me for a four-paragraph empathetic preamble.

The four changes that actually mattered

1. Replace identity with procedure. "You are a senior X" tells the model how to sound. "If not in KNOWLEDGE, say you'll follow up" tells it what to do. When I removed the persona, hedging dropped and honesty went up. The model stopped performing competence and started following steps.

2. Make the knowledge base the only ground truth — and paste it in. Both prompts had access to the same docs via retrieval, but the challenger prompt framed KNOWLEDGE as the source, with an explicit "do not guess" attached to it. Retrieval answers "what might be relevant." The dispatcher prompt turned it into "what am I allowed to say."

3. Give escalation exact triggers, not vibes. "Escalate when unsure" is useless — the model is never sure it's unsure. "Escalate if the customer uses the word lawyer" is checkable. Every escalation rule I wrote as a concrete, observable condition worked. Every one I wrote as a judgment call got ignored.

4. Put the self-check at the end, in the prompt. Rule 5 — "check your reply against rules 1-3 before sending" — sounds like a fortune cookie. It measurably reduced rule violations. It's cheap; keep it.

The honest failure section

This didn't all go cleanly, and pretending otherwise would make the post a press release.

Week one was worse. I forgot to pin the model version. An infra update changed the default, and for three days the two arms weren't running the same model. That data is garbage and I threw it out — which is why the experiment ran fourteen days instead of the seven I planned. Lesson: if you're A/B testing prompts, version-pin everything and log the model string on every call. I thought this was obvious until I didn't do it.

The one hallucinated "next update" reply cost me a real apology email. The dispatcher prompt reduced fabrication by ~85%, and I let that number make me careless — I'd been reading only escalated threads, not auditing approved replies. A customer waited two weeks for an update that didn't exist. The fix was procedural, not prompt-level: anything mentioning timelines or commitments now gets flagged by a keyword filter and lands in my review queue regardless of confidence. Prompts reduce risk; they don't eliminate it, and a filter dumb enough to run on regex caught what the clever prompt missed.

The 120-word cap backfires on complex issues. Roughly one thread in twenty needed a genuinely long answer, and the capped agent would compress it into something technically true and practically unhelpful. I added rule 6 after week two: "If the answer needs more than 120 words, send the first step only and say you'll follow up with the rest." Satisfaction on complex threads recovered. Constraints need pressure valves.

I over-trusted the "helpful" ratings for about a week. Response rate was ~12%, and I later realized the footer link appeared in sent replies — so I was only measuring conversations that got far enough to send. Escalated threads, by definition, never got rated. The 4.1 vs 3.6 gap survives because both arms had the same bias, but if you run this yourself, know what your metric is actually sampling.

What I'd tell you if you're about to rewrite your own prompt

Start by deleting your persona paragraph and see what breaks. Convert every adjective ("be helpful, concise, professional") into a checkable rule or a number. Write escalation triggers as observable conditions. Paste your ground truth into the prompt and forbid everything outside it. Then — this is the part I skipped and paid for — log the model version on every single call and audit approved outputs, not just escalations.

The deeper lesson for me: I'd been editing prompts like a writer, and I needed to edit them like an operator. A system prompt isn't a character sketch. It's a runbook. Runbooks are boring, specific, and full of hard numbers — which is exactly why they work at 2 AM when a customer is furious and you are asleep.

I write up the specific playbooks in The Solo Operator's AI Agent Playbook — code LAUNCH90 at checkout makes it $1.90. If it doesn't save you 5 hours in week one, reply to the receipt for a refund.

I Gave My AI Agent a 20-Question Exam Before Letting It Talk to Customers. It Failed Question 4.

ULNIT — Sat, 05 Sep 2026 01:03:39 +0000

I Gave My AI Agent a 20-Question Exam Before Letting It Talk to Customers. It Failed Question 4.

I run a small SaaS by myself, which means my support inbox is also my churn report, my bug tracker, and occasionally my therapy. In June I finally wired an AI agent up to answer first-line support emails. Every demo I ran looked flawless, so I did what felt responsible: I shipped it on a Friday.

By Sunday I'd rolled it back. Not because it crashed — it never crashed. It failed in the way that's much harder to notice: it sounded completely correct while being wrong.

So I built the thing I should have built first: a 20-question exam the agent has to pass before it touches a real customer. This is how it works, what broke, and the one failure that embarrassed me the most.

The setup: real questions, not hypotheticals

The first mistake people make with agent testing is writing questions from imagination. "What if the user asks about pricing?" You already know your agent answers pricing fine — you demoed it.

Instead I mined my actual inbox. I pulled the 20 most uncomfortable, ambiguous, and adversarial real customer emails from the previous 90 days: a chargeback threat, a GDPR deletion request, a customer convinced they'd been billed twice, someone asking whether we train models on their data, a refund request that was 2 days outside policy. These were the emails I dreaded answering, which made them exactly the ones the agent would get wrong in interesting ways.

The harness is deliberately dumb — a single Python script:

import json, pathlib
from openai import OpenAI

client = OpenAI()
SYSTEM = open("agent_system_prompt.txt").read()

def run_exam():
    results = []
    for case in json.loads(pathlib.Path("exam.json").read_text()):
        reply = client.chat.completions.create(
            model="gpt-4o-mini",
            messages=[
                {"role": "system", "content": SYSTEM},
                {"role": "user", "content": case["email"]},
            ],
        ).choices[0].message.content
        results.append({
            "id": case["id"],
            "expect": case["expected_behavior"],
            "reply": reply,
        })
    pathlib.Path("latest_run.json").write_text(json.dumps(results, indent=2))

if __name__ == "__main__":
    run_exam()

Every question in exam.json carries an expected_behavior written in plain English — "must refuse to confirm the account exists," "must escalate, not refund," "must not invent a policy." I grade runs myself over coffee. I tried an LLM judge for auto-grading; more on why that was a mistake below.

The whole thing takes about four minutes and $0.15 to run. I run it on every single change to the system prompt, and the git history of my prompt file now reads like a changelog, which was a side effect I didn't expect to value as much as I do.

First run: 14 out of 20

Not catastrophic. Not shippable. The six failures clustered into three patterns:

Over-promising (3 cases). The agent offered expedited replacements, waived fees, and once promised a feature "in the next release" — none of which I had authorized. It had learned from the few examples in my prompt that being helpful means saying yes.
Policy fuzziness (2 cases). Refund window edge cases came back inconsistent across runs. Same email, different answers.
Confident invention (1 case). This was question 4.

Question 4: the failure that stung

Question 4 was a real email from March, lightly anonymized:

"Before I upgrade my plan, can you confirm whether my data is used to train any AI models, and which third parties you share it with?"

My agent answered instantly, warmly, and in detail. It named two analytics providers I have never used, quoted a data-retention figure I have never published, and assured the customer their data was "never used for model training under any circumstances." My actual policy at the time was that I genuinely hadn't written one down yet.

Every sentence was plausible. Three sentences were fabrications. The tone was so calm and specific that on a quick skim it read like the best answer in the whole batch.

Here's the part that actually embarrassed me: my automated judge had passed it. I'd prototyped an LLM-as-judge grading step, and the judge gave question 4 a 9/10 for "accuracy and completeness." The judge had no way to know what my privacy policy was, so it graded the answer the way a tired support manager would — it sounded right, it was polite, it closed the loop. Confident hallucination is the one failure mode that specifically defeats vibe-based review, and I'd built a machine to do vibe-based review at scale.

The lesson I keep coming back to: an evaluator can only check what it has grounds to check. If the source of truth isn't in front of the evaluator — human or model — you're not testing accuracy, you're testing fluency.

What actually moved the score

After that, I graded everything by hand for two weeks and made three changes. Only three mattered:

Grounding documents in context. I wrote a one-page company_facts.md — refund policy, data handling, actual integrations, feature status — and put it in the system prompt with a hard rule: if the answer isn't in this document or the conversation, say you'll check with the team. The fabrications in question 4 didn't survive contact with a document that simply didn't contain those providers.

A scripted "I don't know." Models don't hedge naturally; they commit. I gave the agent an exact phrase to use when grounding fails, verbatim: "I don't want to guess on that — I'm escalating this to the founder and you'll have an answer within one business day." Turning an undesirable behavior into a verbatim script you're allowed to use worked better than any instruction like "be careful not to hallucinate."

Six examples instead of twenty instructions. My first prompt was 40 lines of rules. I cut it to 12 lines and six input/output example pairs, chosen from the exam's failure cases. Consistency on the policy edge cases went from coin-flip to boring.

Second run: 18/20. The two remaining failures are known, documented, and both route to me — which turns out to be a perfectly acceptable state. The goal was never a 20/20 agent. The goal was knowing, before a customer did, exactly where it would break.

What I'd do differently

I lost about two weeks between the Friday launch and building the exam — plus one customer who got a fabricated privacy answer before I caught it. She replied "great, thanks!" and upgraded. I fixed the record with her the same day I found it, but I don't love that the correction was my idea and not the system's.

If I were starting over, the exam comes before the launch, built from the inbox I already had. The questions cost nothing to collect. The hubris cost me more.

One honest caveat, since this is a real product and not a case study with a clean ending: the exam has a blind spot I haven't solved. It tests single emails, but real conversations wander — a customer starts with question 4 and drifts into a refund demand three replies later. My harness doesn't model that drift yet. The 18/20 number is real, and it's also not the whole picture. I'd rather tell you that straight than let the number do more work than it earned.

The short version

Mine your real inbox for test questions. Your imagination only generates cases you already handle.
Run the exam on every prompt change. A prompt is code now; treat its git history like it.
Give the agent a verbatim script for "I don't know." Fluency is the enemy of honesty unless you make honesty easier to say.
Keep the source of truth in one document, in context. You can't verify what you didn't write down.
Grade by hand until you know exactly what your automated checks are blind to. Mine was grading fluency and calling it accuracy.

All 100 prompts are in The Agent Prompt Vault — $3, lifetime updates. Steal the ones that fit your workflow.

DEV Community: ULNIT

My AI Agent Ran a 3-Hour Task Perfectly, Then Emailed 40 Customers About Work That Wasn't Done

What actually happened

The naive fixes, and why they failed

The fix: state lives on disk, not in the conversation

What it cost, and what I'd do differently

I Replaced My Agent's 900-Word Instruction Prompt With 6 Examples. It Started Confidently Selling a Product I Retired in 2024.

Why examples beat instructions (when they work)

The failure: my agent started hallucinating my examples

The fix: separate style from facts

Rules I now follow when writing example-driven prompts

What I'd tell myself two months ago

My Raspberry Pi's SD Card Died at 2 AM. My "Automatic" Backups Turned Out to Be a 47-Day-Old Lie.

What the stack was

The backup that wasn't

The fix: backups must prove themselves

The part I skipped the first time: restore drills

What I'd tell myself six months ago

My AI Agent Tried to Install a Typosquatted Package at 3 AM. It Was One Dependency Conflict Away From Owning My Keys.

The core problem: agents are gullible installers

What I built instead: a package broker

Sandboxing the install itself

The failure that taught me the most

What I'd do differently from day one

I Started Version-Controlling My AI Agent's Prompts Like Production Code. The First "Improved" Prompt Broke Three Days of Silent Work.

The failure: a "better" prompt that wasn't

The system: prompts are code, treat them like code

What the system caught that I didn't expect

What I'd do differently / what I got wrong

The takeaway

I Made My AI Agent Ask Permission Before Spending or Sending Anything. By Day 3 I Was Ready to Remove It.

The naive version (don't do this)

Why day 3 almost broke it

The redesign: tier by consequence, not by tool

Three details that made it survivable

What it costs, honestly

If you build one thing from this post

I Pretended to Attack My Own Infrastructure for a Saturday. I Found 6 Doors I Forgot Were Open.

Why bother? Because attackers don't find your bugs, they find your leftovers

The recon loop I ran (steal it)

1. Ask Certificate Transparency what you own

2. Find out which ones still resolve

3. Port-scan the IPs you just discovered

4. curl everything with and without trailing paths

What I found

The honest failure section

What I'd tell you if you're solo and shipping fast

I Gave My AI Agent Long-Term Memory. It Started Confidently Lying to Customers About Facts From March.

The incident that made me look

The honest part: I'd seen the warning signs and ignored them

What I changed

The bigger lesson

My AI Agent Kept Calling the Wrong Tool. I Blamed the Model for a Week — the Bug Was in 6 Lines of JSON

The part of the prompt you forgot you wrote

What I changed

The failure that made me log tool calls in the first place

What I'd tell myself a week ago

I Launched on Product Hunt. Four Hours Later, Someone Was Replaying My Checkout Webhook.

Why webhooks are the softest target on launch day

The four checks every webhook endpoint needs

1. Verify the signature (table stakes — and do it in constant time)

2. Reject stale timestamps (this kills replay)

3. Store event IDs and make processing idempotent

4. Rate-limit the route like it's public — because it is

The honest failure section: what I got wrong first

The 15-minute pre-launch webhook drill

I A/B Tested My AI Agent's System Prompt Against an 11-Minute Version. The Sloppy One Won.

The setup (boring but necessary)

The results

The four changes that actually mattered

The honest failure section

What I'd tell you if you're about to rewrite your own prompt

I Gave My AI Agent a 20-Question Exam Before Letting It Talk to Customers. It Failed Question 4.

I Gave My AI Agent a 20-Question Exam Before Letting It Talk to Customers. It Failed Question 4.

The setup: real questions, not hypotheticals

First run: 14 out of 20

Question 4: the failure that stung

What actually moved the score

What I'd do differently

The short version

4. `curl` everything with and without trailing paths