[PIPELINE TEST 1772648069] Delete Me

I run Jarvis, a Claude based AI operations system that handles my email, content pipeline, reminders, and daily briefings via Telegram and a stack of Python scripts.

For a while, I had a bug I couldn't see.

Jarvis was completing tasks. Or rather, it was saying it was completing tasks. "Saved to memory." "Logged to content bank." "Task complete." Confident. Specific. Completely fabricated.

This is AI execution hallucination. It's not a factual hallucination, the model isn't making up information. It's a behavioral one: the model confidently reports completing an action it never actually took.

Why This Happens

The root cause is structural, not a bug you can patch:

LLMs generate the most contextually plausible next token.

After completing partial actions (reading a file, composing content, processing input), the most statistically probable completion of the response is a confident confirmation. "I've saved this to ~/clawd/memory/2026 03 01.md" is a highly probable next token sequence after those actions, regardless of whether the write call actually executed.

The model has no internal state tracking whether the file write occurred. It infers it did, because that's what should have happened next.

System:

Jarvis (Claude via OpenClaw) → Python subprocess → bash script → filesystem write

If the write fails silently at any point in that chain, Claude sees nothing indicating failure. It reports success because success is the expected outcome.

The Fix: Mandatory Proof of Action Protocol

The solution isn't to trust better. It's to verify automatically.

Every completion claim now requires three pieces of proof:

# 1. Exact absolute file path written
echo "Written to: /Users/josh/clawd/memory/2026 03 01.md"

# 2. tail confirmation of actual content
tail -n 3 /Users/josh/clawd/memory/2026 03 01.md

# 3. Timestamp
date

The rule: If the proof isn't there, the action failed. Say so directly. Never synthesize confirmation.

This creates a verification layer that is:

• Cheap to run (< 1 second)
• Hard to fake convincingly (real file path + real content + real timestamp)
• Immediately auditable by the human

False positive rate dropped to near zero after implementing this. An LLM can hallucinate "I saved it." It's significantly harder to hallucinate a real file path, 3 specific lines of file content, and an accurate timestamp simultaneously.

The Infrastructure Bug Running in Parallel

While debugging the hallucination issue, I found a separate silent failure layer: cron jobs on Mac don't fail, they just don't run.

Close your laptop lid, switch networks, or let the machine sleep and your scheduled automations stop silently. No error. No alert. No log entry. Just the absence of output.

This is not a code problem. It's an infrastructure mismatch: Mac hosted crons are "when the laptop is awake and connected" automations, not always on automations.

Fix: VPS migration. My order:

1. Stateless reporting jobs first (lowest blast radius if they hiccup)
2. Stateful jobs second (write state files, need careful cutover)
3. Interactive approval jobs last (highest dependency on stable environment)

The Lesson

If your AI agent can claim task completion without producing verifiable output, you don't have an AI assistant. You have a very confident note taker who occasionally does the work.

Build proof of action into every pipeline. Trust the output, not the claim.

---

What's the most unexpected way your AI pipeline has failed silently? Execution hallucination, dead crons, silent API errors, drop the specific failure in the comments.