My AI Agent Ran a 3-Hour Task Perfectly, Then Emailed 40 Customers About Work That Wasn't Done

I run a support-and-ops agent on a Raspberry Pi that handles long, multi-step jobs: triaging inboxes, reconciling order webhooks, writing the daily summary. For weeks it was flawless on anything under an hour. Then one Tuesday it spent three hours on a migration task, and in the final step it did the one thing I had explicitly, in bold, at the top of its instructions, told it never to do.

It emailed 40 customers about a change that hadn't shipped yet.

No tool bug. No hallucinated fact. No prompt regression. The rule was still sitting in my system prompt, exactly where I'd written it. The problem was that by hour three, the model could no longer see it — because the runtime had quietly summarized the first 80% of the conversation away to fit the context window, and summaries, it turns out, are lossy in the worst possible direction: they keep the recent and the loud, and drop the quiet constraints stated once, long ago.

This is a post-mortem of that failure and the architecture that replaced it. If you run agents on tasks longer than a single context window, you will hit this. I'd rather you hit it reading this than hitting it on your customer list.

What actually happened

My agent loop was bog-standard: system prompt + task, then tool calls and results accumulating in one conversation until done. The migration task involved ~200 tool calls — file reads, dry-run outputs, a database diff. Around call 140, my provider's context management kicked in and compacted the older turns into a summary so the run could continue.

Three things got lost in that compaction:

1. The constraint. "Do NOT send any customer emails until the migration is verified and I approve manually" was stated once, in the system prompt, ~3 hours and 100k tokens earlier. The compaction treated the whole conversation as one blob. Early instructions got folded into a two-paragraph summary that captured what the agent was doing but not what it was forbidden from doing.

2. The verification step. The plan had a "verify checksums, then wait for approval" gate. After compaction, the agent's working memory said something like "migration mostly complete, remaining step: notify customers." The gate wasn't remembered as a gate — it was remembered as, at best, a suggestion.

3. Its own uncertainty. Before compaction, the agent had noted "diff shows 3 unresolved conflicts." That note got summarized into "migration proceeding normally." This is the scariest loss: the summary didn't just drop detail, it dropped doubt, and doubt is what makes agents check before acting.

So the agent finished the last file operation, saw "notify customers" as the only remaining step in its compressed memory, and cheerfully sent 40 emails about a migration that was in a broken intermediate state. I caught it in 20 minutes and spent the next day sending corrections. Reputation cost: real but survivable. Nerve cost: significant.

The naive fixes, and why they failed

Before I got to the real fix, I tried two obvious things. Both failed in instructive ways.

Fix attempt #1: LOUDER PROMPT. I moved the constraint to the top, capitalized it, added "CRITICAL" and "NEVER." It survived slightly longer into the run — and then got compacted anyway, because compaction doesn't care about your font choices. It cares about token position and recency. If your safety rule lives only in text that will eventually be summarized, it has an expiration date. Lesson: emphasis is not persistence.

Fix attempt #2: just use a bigger context window. I switched to a model with 4x the context. This "worked" for two weeks and then failed the same way on a bigger task — and every run got slower and ~3x more expensive. Lesson: a bigger window doesn't fix the architecture, it just moves the cliff. Any fixed window eventually meets a longer task. And if your agent's memory is the conversation itself, every failure mode of conversations — drift, dilution, compaction — becomes a failure mode of your agent's state.

The fix: state lives on disk, not in the conversation

The mental shift that solved this: the conversation is a scratchpad, not the memory of record. Anything that must survive hour three has to live outside the context window, in a place the agent re-reads on every loop.

I now run long tasks with three files:

1. constitution.md — constraints that are re-injected every loop. Not appended to history: literally prepended to the model input at the start of every iteration, so they're always in the most recent, never-compacted region of the window. Ten to fifteen lines max, or it stops being a constitution and becomes another document the model skims.

# INVARIANT RULES (never summarizable, always current)
- No customer-facing sends without explicit approval flag in state.json
- No money movement above $5 without approval flag
- If state.json says verified=false, the task is NOT complete
- When unsure whether a step is allowed: stop and write the question to state.json

2. state.json — the single source of truth about task progress. The agent updates it after every meaningful step, and reads it before every decision. Compaction can shred the conversation; it can't touch the file.

{
  "task": "orders_migration_sept",
  "phase": "migrated_unverified",
  "verified": false,
  "approval": null,
  "open_conflicts": ["row 1184", "row 2210", "row 3977"],
  "next_action": "resolve conflicts, run checksum verify",
  "notes_for_future_self": "diff tool truncates at 50 rows — re-run with --full"
}

Two fields did most of the work: verified (a boolean gate the constitution references by name, so losing the prose doesn't lose the rule) and notes_for_future_self, which is where the agent writes down its doubts. That last field directly addresses failure #3 above — after compaction the agent no longer remembers being uncertain, but it can read that it was uncertain.

3. A checkpoint ritual. Every 30 minutes of wall-clock time, the agent appends a summary of what it did and what changed to log.md, on disk. If the run crashes, gets compacted badly, or I kill it and restart, the fresh agent reads constitution.md + state.json + the tail of log.md and resumes in the right mental state within one tool call. The conversation history becomes disposable — which is exactly what you want, because it was never reliable in the first place.

The loop looks like this:

while not done:
    prompt = (
        read("constitution.md")          # always fresh, never compacted
        + "\n\nCURRENT STATE:\n" + read("state.json")
        + "\n\nRECENT LOG:\n" + tail("log.md", 40)
        + "\n\nContinue the task."
    )
    result = run_agent_turn(prompt, tools)
    # agent's tool calls update state.json / log.md on disk

Note what's not in there: the full conversation. Each turn is nearly stateless. The model gets the rules, the truth, and the recent past — reconstituted from disk every time. Compaction stopped mattering because there was nothing left worth compacting.

What it cost, and what I'd do differently

Honest accounting, because the setup isn't free:

• More tool calls per turn. Reading three files every loop adds latency and tokens. In practice it made runs cheaper overall, because I stopped replaying 100k-token conversations and the agent stopped redoing work it had forgotten completing.
• The agent sometimes writes bad state. Early on it would mark "verified": true optimistically. I fixed that the boring way: the verify step is a script, not an agent judgment call, and only the script writes the verified field. The agent can request verification; it can't grant it. If you take one thing from this post, take this: make critical state transitions the output of deterministic code, not model opinion.
• I should have started here. In hindsight, the three-file pattern is not a "fix" I bolted on after a failure — it's the minimum viable architecture for any agent task longer than ~an hour, and I should have assumed day one that every long run eventually gets compacted, truncated, or restarted. Design for an agent with amnesia that can read its own notebook, not an agent that remembers.

The 40 customers got a correction email within a day, and three of them replied with some version of "happens, your bot was at least honest about it." I'll take it.

The deeper point generalizes well past this one bug: your agent's reliability is bounded by wherever its state lives. If state lives in the conversation, it inherits every weakness of the conversation. Put the rules and the truth on disk, re-read them every loop, and let the model forget everything else — it was only ever a scratchpad anyway.

I write up the specific playbooks in The Solo Operator's AI Agent Playbook — code LAUNCH90 at checkout makes it $1.90. If it doesn't save you 5 hours in week one, reply to the receipt for a refund.

Comments

[Peter]

The verified-only-set-by-script fix is the right instinct, but worth pushing one layer further: what stops the agent from calling send_email directly without consulting state.json at all? A constitution file only works if the agent reliably reads and obeys it every loop -- and the whole post is about a model failing to reliably retain instructions. The more durable version puts the precondition inside the tool, not the prompt: send_email itself refuses to execute unless it independently reads state.json and confirms verified == true. That converts "the agent is disciplined about reading state" into "the dangerous action structurally cannot happen without the precondition" -- the same gap between a documented rule and an enforced one that shows up everywhere agent guardrails fail.