My AI Agent Reported Success for 9 Straight Days. Then a Customer Told Me It Had Been Failing All Along.

Nine days ago, my support agent started its nightly run at 2:00 AM, same as always. At 2:14 AM it logged processed 29 emails, 0 errors. Same the next night. And the night after that.

For nine straight nights, that agent never missed a run, never threw an error, never triggered a retry. By every metric I was tracking, it was the most reliable piece of infrastructure in my one-person business.

Then a customer emailed me personally: "Why did your support team tell me to check the docs for a product I never bought? Twice?"

I pulled the logs. Exit code 0, all green. Then I pulled the agent's actual output and read it — really read it — for the first time in over a week. My stomach dropped.

This is the story of how a "healthy" agent quietly failed for nine days, why none of my monitoring caught it, and the specific systems I built so it can't happen again.

What the agent was supposed to do

I run a solo operation, so support volume is small but constant. Every night at 2 AM, an agent on my Raspberry Pi:

1. Fetches new support emails via my email provider's API
2. Classifies each one (refund request, bug report, pre-sale question, etc.) using an LLM call
3. Drafts a reply for anything routine, and routes anything sensitive — refunds, angry customers, legal-sounding language — to a human queue (me)
4. Logs a summary to a dashboard I glance at over coffee

It had been running for about two months and had genuinely earned its place. I trusted it. That trust was the problem.

What it was actually doing

Here's what I found when I dug into those nine nights of output.

Around night one of the failure window, my LLM API key hit a spending limit I'd forgotten to raise. Every classification call started returning a 429.

If the agent had crashed, I'd have known — I get notified on failed runs. But it didn't crash. I had written a fallback, the way you do when you're being "defensive":

try:
    label = classify(email)
except Exception:
    label = "general"  # safe default, I told myself

So every email for nine nights got labeled general. And general was on the auto-reply list. Which means every single email — refund requests, bug reports, a furious customer on night four, two pre-sale questions that were basically money on the table — got the same bland template reply: "Thanks for reaching out! Here are some resources that might help..."

253 emails. All "handled." Zero errors logged.

The worst part isn't the code. It's that I built the dashboard to show me run status and volume, because those are easy to graph. "Processed 29 emails" feels like progress. Nobody graphs "percentage of emails labeled general," because you don't expect to need to.

The part I'm embarrassed about

I spent the next morning writing 31 personal apology emails. Some customers had replied to the template and gotten another template. One of the pre-sale questions — a team lead asking about volume licensing — had gone silent after my bot's brush-off. I got that one back with a real conversation, but I don't know how many I didn't get back.

The uncomfortable lesson: exit code 0 is not a health signal. An agent that always succeeds is not reliable — it's suspicious. Crashes are honest. Silent fallbacks are not. My agent hadn't been handling support for nine days. It had been performing support, to an audience of one dashboard that only checked whether the show started on time.

What I fixed (the system, not just the bug)

Raising the spending limit took thirty seconds. That was the easy part. The real work was making this class of failure impossible to miss. Here's what actually went in:

1. Fallbacks are incidents, not features.

The silent default is gone. Now any fallback path increments a degradation counter and routes output to a human instead of to the customer:

try:
    label = classify(email)
except Exception as e:
    stats["degraded"] += 1
    logger.error("classify failed for %s: %s", email.id, e)
    label = "NEEDS_REVIEW"  # goes to me, never to the customer

If more than 10% of a run is degraded, the run is marked failed and I get pinged. A fallback should feel like a bruise, not a cushion.

2. Log decisions, not just events.

The nightly summary now includes the label distribution: refund: 3, bug: 7, general: 19. That single line would have caught it on night two — a support inbox that's suddenly 100% "general" is not an inbox, it's a warning.

3. Canary items.

Every night, before the real run, the agent processes three known test emails I control: one refund request, one bug report, one pre-sale question. If any comes back mislabeled, the run aborts and pages me:

CANARIES = [
    ("canary-refund-001", "refund"),
    ("canary-bug-002", "bug_report"),
    ("canary-pricing-003", "sales"),
]

def check_canaries(results):
    for cid, expected in CANARIES:
        if results.get(cid) != expected:
            raise RuntimeError(f"Canary failed: {cid} "
                               f"(expected {expected}, got {results.get(cid)})")

This catches broken classification even when every real email would have been mislabeled identically. It's the cheapest insurance I've ever built.

4. A distribution tripwire.

A cron check compares label distributions across nights. If any label jumps above 80% of volume two nights in a row, I get an email. Weird distributions are almost never about your customers; they're about something that changed under you — an API, a key, a selector, a quota.

5. Two human minutes, every morning.

No automation replaces this one. With my coffee, I read five actual outputs from last night's run. Not a summary — the outputs. It takes two minutes, and it's the only check that would have caught this on day one. Sampling beats dashboards, because dashboards only show you what you thought to ask about.

The checklist I run before trusting any new agent

After this, I don't ship an agent — support, recon, content, anything — until it passes five questions:

• What does it do when a dependency fails? (If the answer is "fall back silently," it doesn't ship.)
• Can I see its decisions, not just its exit codes?
• Does it have canaries — known inputs with known-correct outputs?
• What does "weird" look like for this agent, and who notices?
• Am I personally reading a sample of its output, on a schedule?

Five questions. Nine days of damage would have cost me roughly twenty minutes to prevent. That math is why I'm writing this down instead of quietly patching it and moving on.

If you run agents unattended — overnight jobs, Pi fleets, anything that works while you sleep — go read five of last night's outputs today. Not the logs. The outputs. I hope yours are boring. Mine weren't, and now my monitoring is.

---

I write up the specific playbooks in The Solo Operator's AI Agent Playbook — code LAUNCH90 at checkout makes it $1.90. If it doesn't save you 5 hours in week one, reply to the receipt for a refund.