The cost of finding a failure after the customer finds it

Originally published on Loop & Retry — field notes on building LLM agents that survive production.

A customer emails support: "I paid for your service three weeks ago but my account still says 'pending.' I haven't been able to log in once."

Your on-call engineer investigates. It takes 30 minutes to trace the account through five systems, pull logs from three different services, and understand that the agent that was supposed to activate the account failed silently on day 1 of those three weeks.

The failure itself happened instantly. The cost was paid over the next 21 days.

The timeline of a late-discovered failure

Let's map the cost curve:

Day 0: Agent fails silently (1 minute, undetected)
├── Operation fails at step 3 of 5
├── No alert fires
├── State is partially written (customer in auth system, not in feature system)
└── Cost so far: $0.01 (the failed operation)

Days 1–20: Silence
├── Customer tries to log in, sees "pending"
├── No alert fires (no health check noticed)
├── No monitoring surfaced the inconsistency
├── Cleanup cost accumulating: $0
└── Cost so far: $0.01 (still just the operation)

Day 21: Customer complains
├── Support ticket created (human context-switch, email read, ticket filed): $2
├── On-call engineer paged (or assigned to backlog): $5
├── Investigation begins: pull logs, query systems, understand the problem: $30
├── Time spent: 30–45 minutes
└── Cost so far: $37

Day 21: Incident response
├── Decision: rollback vs. forward-fix? (engineer judgment call, could be wrong): $10
├── Implementation: write a script, test it, run it: $20
├── Communication: notify customer, explain delay: $5
└── Total investigation + fix cost: $72

Days 22–40: Aftermath
├── Customer follow-ups (slower support responses due to backlog): $5
├── Root cause analysis (why wasn't this caught?): $30
├── New monitoring/alerting deployed: $50
└── Total: $155

**Total cost: $192 for one account**

If you run this flow 100 times (fleet of 1,000 agents, 10% failure rate), you're paying $19,200 in human time for failures that could have been detected and fixed automatically.

Why detection latency multiplies cost

The core issue: a problem you discover yourself is cheap to fix. A problem a customer discovers is expensive to fix.

Cost factor 1: Triage delay

When an alert fires at 3am, an engineer (or automation) immediately checks the logs. When a customer emails, the failure sits in a queue until someone reads the support ticket. That's anywhere from 15 minutes to 24 hours of additional latency.

In that delay, the problem persists. For an account that's stuck "pending," the customer can't use the service. For a data pipeline that failed to update, downstream jobs operate on stale data. The damage compounds.

Cost factor 2: Investigation without context

An automated alert fires with structured context:

{
  "alert": "orphaned_account_state",
  "account_id": "acct_xyz789",
  "missing_from": ["feature_service"],
  "exists_in": ["auth_system", "billing"],
  "detected_at": "2026-08-17T03:15:23Z",
  "last_write_timestamp": "2026-08-16T17:43:01Z",
  "failure_type": "service_timeout"
}

A customer complaint gives you:

I signed up three weeks ago and it still doesn't work. Can you help?

Now the engineer has to:

• Search logs by timestamp (when was the signup?)
• Cross-reference account ID with multiple systems
• Reconstruct what happened by reading traces
• Infer which step failed based on partial state
• Decide what "broken" means (should it be active? refunded? re-run?)

Investigation time: 30–60 minutes instead of 2 minutes. 15–30x cost multiplier.

Cost factor 3: Wrong fix decisions

Without automated alerts, the engineer is flying blind. They have to guess at the root cause and severity based on incomplete information.

• Is this account the only one? Or are there 50 like it?
• Did this happen once, or is it still happening?
• Is it a temporary blip, or a systemic issue?
• Should we rollback the account, or will that cause data loss?

With an automated detector, you know:

Orphaned account detection (hourly run):
├── Found 17 accounts in auth_system but not feature_service
├── All signed up between 2026-08-15 and 2026-08-17
├── Failure pattern: all failed on feature_service.enable() with "connection_timeout"
├── Recommendation: feature_service connection pool is exhausted
└── Action: re-enable accounts via feature_service.enable() (idempotent, safe)

Now the fix is automatic (or a one-line command), the scope is known, and the risk is understood. Cost: 1 minute. Without automation, cost: 30 minutes + risk of the wrong fix.

The detection strategies and their costs

You have three choices:

Strategy 1: Manual discovery (customer complains)

Detection latency: Hours to days

Detection cost: $0 (unpaid human labor)

Fix cost: $200–500 per incident

Total cost per failure: $200–500

This is the default if you don't build anything. The cost is paid by customers and support staff, not by engineering.

Strategy 2: Reactive monitoring (metrics, logs, dashboards)

Detection latency: Minutes to hours (depends on engineer reviewing dashboards)

Detection cost: ~$50/month for infrastructure (CloudWatch, DataDog, etc.)

Fix cost: $50–150 per incident (engineer has structured data to work from)

Total cost per failure: $50–150 per incident + infrastructure cost

Example: Engineer is on-call, gets a Slack alert that "account-activation failure rate jumped from 0.01% to 5%." Engineer checks the dashboard, sees a spike in service_timeout errors in the feature service, checks service health, finds the issue, and fixes it. Total time: 10 minutes.

This works if:

• You have visibility into the right metrics
• Someone is actively monitoring them (or alerts are well-tuned)
• The alert is specific enough to guide the fix

Strategy 3: Proactive health checks (synthetic, detector-driven)

Detection latency: Minutes (same cadence as the check)

Detection cost: ~$100–500/month (infrastructure + engineering to build detectors)

Fix cost: $5–50 per incident (automation does most of the work, engineer confirms)

Total cost per failure: $5–50 per incident + infrastructure cost

Example: A health check runs every 5 minutes and asks: "Are there customers in auth but not features?" If yes, it fires an alert with the exact list of affected IDs and a recommended fix script. An on-call engineer gets the alert, reviews the list, runs the fix, done. Total time: 5 minutes (and the next iteration, this could be fully automated).

This works if:

• You can define "healthy" state in code (hard for complex systems)
• The detector is fast enough to catch issues early
• You actually act on the alert (detector fatigue is a real cost)

When detection costs more than the failure itself

Here's the catch: building a detector costs engineering time. For a low-frequency failure, that cost isn't justified.

Low-frequency failure (happens once a month):
├── Detector cost: $200 (2 hours of engineering to build + maintain)
├── Fix cost without detector: $300 (one customer discovery)
├── Total cost: $500
│
└── If we build detector:
    ├── Detector cost: $200
    ├── Detection cost: ~$5/month in compute
    ├── Fix cost per incident: $20 (automatic, mostly)
    ├── Break-even: first incident + one month of compute
    └── But only if the failure happens again

If the failure is truly one-off, the detector wasn't worth it. The problem is you don't know the frequency until after it fails.

The cost calculus: which failures are worth detecting

Ask three questions:

1. How frequent is the failure? (daily, weekly, monthly, one-time)
2. How many customers does it affect at once? (one, tens, thousands)
3. How expensive is manual discovery? (10 minutes, 1 hour, cross-team incident)

Frequent + wide-impact + expensive = always detect

Example: "Agent fails to activate account" in a high-volume signup system.

• Frequency: Happens once a day due to transient service timeouts
• Impact: 10–100 customers per incident
• Discovery cost: $200–500 (each customer posts support ticket, takes time to triage)
• Detector cost: $300 (2 hours to build health check)
• Break-even: One incident. You detect it automatically, notify the team, and they fix the root cause (service timeout). Paid for itself.

Build the detector. The math is brutal without it.

Rare + narrow-impact + cheap = don't detect

Example: "Webhook signature validation fails for one specific customer."

• Frequency: Once every few months
• Impact: One customer, reproducible on their end
• Discovery cost: $30 (customer reports it, engineer checks the integration)
• Detector cost: $200 (would need integration-level testing)
• Break-even: Never, unless it starts happening frequently

Don't build the detector. The customer can report it, you fix it, and that's cheaper than proactive monitoring.

Medium frequency + medium impact + medium cost = measure the tail

Example: "Feature flag evaluation fails, users see default behavior"

• Frequency: Once a week
• Impact: 1–5 users per incident (they see default, not personalized experience)
• Discovery cost: $50 (user notices, files a support ticket, engineer investigates)
• Detector cost: $400 (need to instrument flag evaluations, set up alerts)
• Break-even: 8–10 incidents

If you have a reasonable prediction that it'll happen more than 10 times, build the detector. If you're not sure, measure the current cost (how often does this actually create support tickets?) before building.

What to detect first

Start with the failures that:

1. Create orphaned or inconsistent state. These are expensive to fix manually because someone has to find the orphaned records and decide how to clean them up.
2. Have a clear, testable "healthy" state. "Customers should exist in all three systems" is testable. "The user had the right experience" is not.
3. Affect multiple customers at once. A failure that hits 50 people is worth detecting faster than a failure that hits one person.
4. Have low false-positive rates. A detector that fires 100 times and 99 are false alarms will be ignored. Start with high-confidence signals.

The early-warning pattern

If you can't build a perfect detector, build an early-warning detector instead:

def early_warning_orphaned_accounts():
    """
    Find accounts in the auth system but not activated
    in the feature service within 24 hours of signup.

    This is an early warning—they might activate later,
    but if the system is healthy, they should activate
    within an hour.
    """
    recent_auth_signups = auth.list_accounts(
        created_after=now() - timedelta(hours=24),
        status="pending_activation"
    )

    not_in_features = [
        acc for acc in recent_auth_signups
        if not features.is_active(acc.id)
    ]

    if not_in_features:
        # Fire a low-urgency alert; may be false positives
        # But if it happens 50 times, something is wrong
        alerts.warn(
            "orphaned_accounts_24h",
            count=len(not_in_features),
            account_ids=[a.id for a in not_in_features]
        )

This doesn't guarantee a failure—accounts might activate later. But if you see this alert repeatedly, you know something is slower or broken. The detection cost is minimal (one query per hour), but it gives you early visibility.

What I'd do

1. List your most expensive incidents from the last year. Which ones took the longest to discover? Which ones affected the most customers? Which ones required the most manual cleanup?

2. For each expensive incident, ask: could an automated detector have surfaced this 1 hour earlier? If yes, estimate the time/cost savings.

3. Build detectors for the top 3 incidents by potential cost savings. Expect each to take 4–6 hours of engineering. Validate that you save more than 4–6 hours per incident when they recur.

4. Start with "early warning" detectors before perfect ones. A detector that fires 10 times and catches one real problem is better than a perfect detector that takes 3 weeks to build and launches after the next incident.

5. Measure detector precision and recall. How many alerts fire? How many are real problems? If precision drops below 50%, tune the alert. Detector fatigue is a silent cost.

6. Automate the fix when you can, alert when you can't. If the health check finds orphaned accounts and the fix is clear (re-run the activation step), automate it. If the fix requires judgment, alert and let humans decide.

The arithmetic is simple: a failure you detect at 1am (or via automation) costs 10x less to fix than a failure a customer discovers at 9am. That's a 10x leverage point. Most teams ignore it because the cost is hidden—it's paid in support load and customer frustration, not in the LLM spend that engineering can see.

But if you run a fleet, the math compounds. 100 agents, 1% failure rate, 21-day detection latency on average: you're paying $10,000+ per month in hidden failure-discovery costs. A $500 investment in detectors pays for itself in a week.

The detection latency numbers here are from SaaS incidents I've traced—mostly in onboarding and feature activation flows. Your latency may be shorter (if you have good alerts) or longer (if customers don't complain immediately). The principle holds: the cost of finding a failure jumps discontinuously the moment a customer finds it first.

ref: phase-2-content-2026-08-17