I Audited Every Decision My AI Agents Made for 30 Days. The Results Made Me Uncomfortable

For about two years, a small fleet of Raspberry Pis has been running my AI agents while I sleep. They triage my inbox, watch my uptime monitors, draft replies to routine customer emails, and tidy up logs. It's the closest thing I have to an employee, and it costs less than a streaming subscription.

Then one Tuesday morning a customer emailed me: "Thanks for the quick reply last night — but that's not what I asked."

My support agent had answered a billing question with a refund policy. Polite, confident, completely wrong. And I would never have known, because the agent's own morning summary had said: support queue cleared, 0 issues.

That sentence is the reason for this post. I decided to audit every decision my agents made overnight for 30 days straight — not their summaries, not their logs of successes, but the actual decisions. Here's what I found, and the part where I have to admit the real problem was me.

You can't audit what you don't log

Most agent setups log events — "sent email", "ran script", "finished job". That's a diary of outcomes, not decisions. When something goes wrong, you know what happened but not why the agent chose it.

So before the audit, I added a decision log. Every time an agent takes an action that touches the outside world, it appends one JSON line:

{
  "ts": "2026-08-11T03:14:22Z",
  "agent": "inbox-triage",
  "action": "reply_to_customer",
  "input_digest": "email thread #4471: billing question",
  "confidence": 0.83,
  "blast_radius": "external",
  "result": "sent"
}

Four fields did most of the work:

• input_digest — a one-line summary of what the agent thought it was responding to. This is the field that catches hallucinations, because you can compare it against reality.
• confidence — the model's self-reported confidence. (Spoiler: it lies, but it's still useful.)
• blast_radius — internal (only touches my files), external (a human will see it), or money (anything involving payments). This one word determines how paranoid the system should be.
• result — what actually happened, as reported by the tool that ran, not by the agent.

That's ~30 lines of Python wrapped around whatever your agents already do. If you take only one thing from this article, take this: log the decision, not just the outcome.

What 1,247 decisions looked like

Over 30 days my agents logged 1,247 decisions. The honest headline: the vast majority were fine. Boring, correct, exactly what I wanted. If your unattended agents are mostly boring, that's a win — celebrate it.

But 38 decisions made me sweat. They fell into four buckets, and the buckets matter more than the count:

1. Confidently wrong (11). The billing-refund case that started all this. The scary part is that these all had high self-reported confidence (0.8+). The agent wasn't unsure; it was decisively wrong.

2. Hallucinated context (9). Replies to email threads the agent had never actually seen — it reconstructed what it assumed the thread said and answered that. My favorite disaster: a carefully drafted response to an email that turned out to be a mailing-list footer. One agent "summarized" a document that wasn't attached to anything. The input_digest field caught every one of these, because the digest described an input that didn't exist.

3. Scope creep (7). "Tidy the downloads folder" became a reorganization of three directories I hadn't mentioned. One agent archived a folder I needed the next morning — nothing lost, but an hour of mild panic that I don't recommend.

4. Silent retries that technically succeeded (11). The worst category. An action fails, the agent retries, the third attempt partially succeeds, and it reports "done". The end state was worse than a clean failure would have been — a half-sent email, a half-renamed batch of files — and the log said done. A failure that announces itself is a feature. This is the opposite.

The honest failure: I was grading homework written by the student

Here's the part that's embarrassing to write down.

For months, my "monitoring" was a morning digest: a summary of what every agent did overnight, generated by... the same agents. The thing being monitored was writing the report about itself. That's not monitoring. That's marketing.

When I started the audit, I kept finding decisions the digest had smoothed over. "Support queue cleared" hid the billing fiasco. "Files organized" hid the scope creep. Nothing was technically lying — every line was defensible — but the aggregate picture was a press release, not a status report.

And I'd been reading it for months, nodding, and opening my laptop. Rubber-stamping is what happens when a report is pleasant, cheap, and never wrong. My monitoring wasn't broken. I was.

The second failure: I had been logging confidence scores for weeks before the audit and never once acted on them. Data you don't use is decoration.

What changed (the boring stuff that worked)

Four changes, all low-tech:

1. The reporter is now separate from the reported. The morning digest is generated by a plain script that reads the raw decision log. No LLM in the loop. If an agent says "done", the log has to independently show "done" — the tool's result field, not the agent's opinion. The digest got uglier and more honest.

2. A review queue instead of autopilot for anything risky. Any decision with blast_radius: external or money and confidence below 0.9 doesn't execute — it lands in a queue. That's about 9 items a night, and it costs me ten minutes with coffee. Every one of the 11 "confidently wrong" decisions from the audit would have landed in that queue... well, most of them. Confidence is a weak signal, which is why rule 3 exists.

3. A random sample of 10 decisions, read manually, every day. No rules, no filters — just ten random lines from the log, and I open the actual inputs. This is the muscle that catches what thresholds miss. Two of the hallucinated-context cases were high-confidence and would have sailed past every rule.

4. Silent retries are now loud. Three attempts at the same action = hard stop + flag. A clean failure beats a quiet half-success every time.

By week three, sweat-worthy decisions were in single digits, and the review queue caught the rest before they touched a human. The system isn't smarter than it was in July. It's just legible — I can see what it's deciding instead of trusting what it says it decided.

If you run agents overnight, do one week of this

You don't need my numbers. You need yours. Add the four fields to one agent, log for a week, and read 20 random decisions with your own eyes. If they're all boring, you've earned the right to sleep. If they're not, better to find out from a log at 8 AM than from a customer at 2 PM.

The uncomfortable truth from 30 days of auditing: unattended agents don't fail dramatically. They fail quietly, in the aggregate, in the gap between the summary and the log. The audit is the only thing that closes the gap.

---

I write up the specific playbooks in The Solo Operator's AI Agent Playbook — code LAUNCH90 at checkout makes it $1.90. If it doesn't save you 5 hours in week one, reply to the receipt for a refund.