The cost of finding a failure after the customer finds it

Originally published on Loop & Retry — field notes on building LLM agents that survive production.

A customer emails support: "I paid for your service three weeks ago but my account still says 'pending.' I haven't been able to log in once."

Your on-call engineer investigates. It takes 30 minutes to trace the account through five systems, pull logs from three different services, and understand that the agent that was supposed to activate the account failed silently on day 1 of those three weeks.

The failure itself happened instantly. The cost was paid over the next 21 days.

The timeline of a late-discovered failure

Let's map the cost curve:

Day 0: Agent fails silently (1 minute, undetected)
├── Operation fails at step 3 of 5
├── No alert fires
├── State is partially written (customer in auth system, not in feature system)
└── Cost so far: $0.01 (the failed operation)

Days 1–20: Silence
├── Customer tries to log in, sees "pending"
├── No alert fires (no health check noticed)
├── No monitoring surfaced the inconsistency
├── Cleanup cost accumulating: $0
└── Cost so far: $0.01 (still just the operation)

Day 21: Customer complains
├── Support ticket created (human context-switch, email read, ticket filed): $2
├── On-call engineer paged (or assigned to backlog): $5
├── Investigation begins: pull logs, query systems, understand the problem: $30
├── Time spent: 30–45 minutes
└── Cost so far: $37

Day 21: Incident response
├── Decision: rollback vs. forward-fix? (engineer judgment call, could be wrong): $10
├── Implementation: write a script, test it, run it: $20
├── Communication: notify customer, explain delay: $5
└── Total investigation + fix cost: $72

Days 22–40: Aftermath
├── Customer follow-ups (slower support responses due to backlog): $5
├── Root cause analysis (why wasn't this caught?): $30
├── New monitoring/alerting deployed: $50
└── Total: $155

**Total cost: $192 for one account**

If you run this flow 100 times (fleet of 1,000 agents, 10% failure rate), you're paying $19,200 in human time for failures that could have been detected and fixed automatically.

Why detection latency multiplies cost

The core issue: a problem you discover yourself is cheap to fix. A problem a customer discovers is expensive to fix.

Cost factor 1: Triage delay

When an alert fires at 3am, an engineer (or automation) immediately checks the logs. When a customer emails, the failure sits in a queue until someone reads the support ticket. That's anywhere from 15 minutes to 24 hours of additional latency.

In that delay, the problem persists. For an account that's stuck "pending," the customer can't use the service. For a data pipeline that failed to update, downstream jobs operate on stale data. The damage compounds.

Cost factor 2: Investigation without context

An automated alert fires with structured context:

{
  "alert": "orphaned_account_state",
  "account_id": "acct_xyz789",
  "missing_from": ["feature_service"],
  "exists_in": ["auth_system", "billing"],
  "detected_at": "2026-08-17T03:15:23Z",
  "last_write_timestamp": "2026-08-16T17:43:01Z",
  "failure_type": "service_timeout"
}

A customer complaint gives you:

I signed up three weeks ago and it still doesn't work. Can you help?

Now the engineer has to:

• Search logs by timestamp (when was the signup?)
• Cross-reference account ID with multiple systems
• Reconstruct what happened by reading traces
• Infer which step failed based on partial state
• Decide what "broken" means (should it be active? refunded? re-run?)

Investigation time: 30–60 minutes instead of 2 minutes. 15–30x cost multiplier.

Cost factor 3: Wrong fix decisions

Without automated alerts, the engineer is flying blind. They have to guess at the root cause and severity based on incomplete information.

• Is this account the only one? Or are there 50 like it?
• Did this happen once, or is it still happening?
• Is it a temporary blip, or a systemic issue?
• Should we rollback the account, or will that cause data loss?

With an automated detector, you know:

Orphaned account detection (hourly run):
├── Found 17 accounts in auth_system but not feature_service
├── All signed up between 2026-08-15 and 2026-08-17
├── Failure pattern: all failed on feature_service.enable() with "connection_timeout"
├── Recommendation: feature_service connection pool is exhausted
└── Action: re-enable accounts via feature_service.enable() (idempotent, safe)

Now the fix is automatic (or a one-line command), the scope is known, and the risk is understood. Cost: 1 minute. Without automation, cost: 30 minutes + risk of the wrong fix.

---

Continue reading: the rest of this post — including the sections that follow — is on Loop & Retry: read the full article