Measuring retry success: the metric that tells you if retries work

Originally published on Loop & Retry — field notes on building LLM agents that survive production.

The fleet-patterns post explained how to cap retry costs across a fleet. The observability post covered idempotency keys and correlation. But you still can't answer the question that matters in production: are retries working, or are they just making things slower and more expensive?

A retry either recovers a transient failure or burns budget on a permanent one. Without visibility into which is happening, you're flying blind — you can't tell if your retry budget is too generous (wasting money) or too stingy (giving up too early), and you can't tell if a retry storm is fixing an outage or amplifying it.

This post is about three metrics that make retries visible, and the instrumentation pattern that produces them.

The blind spot

Most production systems log retry attempts, but they log them as events: "attempt 1 failed, attempt 2 succeeded" or "attempt 3: gave up." That's useful for reconstructing a specific failure after the fact, but it doesn't tell you about the pattern across a fleet.

Consider a real scenario: one service starts failing. Its clients see errors and begin retrying. Some of those retries succeed — the service recovers momentarily — but then it fails again. One client retrying a few times is reasonable. Twelve clients each retrying a few times is a retry storm. Hundreds of clients each retrying is a cascade.

How do you know which one is happening? You count. But most monitoring systems don't emit "how many retries happened, and how many of them succeeded" as a metric — they log individual events that aggregation tools have to scrape together later. By then it's too late to alert on it in real-time.

Three metrics that matter

Metric 1: retry success rate (the one that steers everything else)

This is the fraction of retried operations that ultimately succeeded, measured per operation type (or per dependency, depending on your architecture). High success rate (>70%) means retries are earning their cost — most of the time they're fixing transient failures, not burning budget. Low success rate (<20%) means you're likely hitting permanent failures repeatedly.

import dataclasses
from collections import defaultdict

@dataclasses.dataclass
class RetryMetrics:
    operation_name: str
    total_retried: int = 0          # operations we retried at least once
    retried_succeeded: int = 0      # of those, how many eventually succeeded
    retried_failed: int = 0         # of those, how many exhausted retries
    total_attempts_consumed: int = 0  # across all retried ops

def success_rate(metrics: RetryMetrics) -> float | None:
    if metrics.total_retried == 0:
        return None
    return metrics.retried_succeeded / metrics.total_retried

# Example from production:
auth_retries = RetryMetrics(
    operation_name="token_refresh",
    total_retried=145,
    retried_succeeded=102,
    retried_failed=43,
    total_attempts_consumed=312  # 102*2 (mostly 1 retry each) + 43*3 (gave up after 3)
)

print(f"Token refresh success rate: {success_rate(auth_retries):.1%}")
# Output: Token refresh success rate: 70.3%

This single number steers everything downstream. If it's high, your retry logic is working as intended. If it drops suddenly, something structural has broken (a service misconfiguration, a network partition, a permanent bug). If it's stubbornly low, you're retrying operations you shouldn't be retrying — time to audit your error classification.

Metric 2: retry amplification factor

This is the ratio of total attempts made to unique operations attempted, measured per class of failure. A factor of 1.0 means no retries happened (all operations succeeded on the first try). A factor of 1.5 means on average each operation was attempted 1.5 times. A factor of 3.0 means each operation was attempted three times on average — a strong signal of a retry storm.

def amplification_factor(metrics: RetryMetrics) -> float:
    if metrics.total_retried == 0:
        return 1.0  # no retries = factor of 1
    # Average attempts per retried operation
    return metrics.total_attempts_consumed / metrics.total_retried

# Continuing the token refresh example:
print(f"Amplification factor: {amplification_factor(auth_retries):.2f}x")
# Output: Amplification factor: 2.15x
# Interpretation: 70% succeeded on second try, 30% needed 3+ attempts before failing

The amplification factor is your early warning system. When it spikes without a corresponding success-rate drop, it means a failure is either:

1. Becoming steadily permanent (more retries needed per operation) — suggests a degraded dependency
2. Becoming intermittent in a pathological way (lots of oscillation) — suggests a resource exhaustion or cascading failure

Combine it with the success rate: high amplification + low success = major incident. High amplification + high success = your retries are working hard but earning it. Low amplification + high success = boring and healthy.

Metric 3: retry latency percentile impact

This is the p50, p95, p99 increase in operation latency for operations that were retried versus those that succeeded on the first attempt. It tells you the time cost of retrying, which is often invisible in cost accounting (you count the token cost but forget that the user was waiting).

import statistics

@dataclasses.dataclass
class PercentileMetrics:
    operation_name: str
    first_try_latencies_ms: list[float]  # operations that succeeded immediately
    retried_latencies_ms: list[float]    # operations that needed retries

def latency_percentile_cost(pctile_metrics: PercentileMetrics):
    for p in [50, 95, 99]:
        first_try = statistics.quantiles(pctile_metrics.first_try_latencies_ms, n=100)[p-1]
        retried = statistics.quantiles(pctile_metrics.retried_latencies_ms, n=100)[p-1]
        overhead = retried - first_try
        print(f"p{p} latency: +{overhead:.0f}ms for retried operations ({retried:.0f}ms vs {first_try:.0f}ms)")

# Example:
# p50 latency: +120ms for retried operations (350ms vs 230ms)
# p95 latency: +890ms for retried operations (2100ms vs 1210ms)
# p99 latency: +3200ms for retried operations (4800ms vs 1600ms)

This matters because latency is a user-facing cost. A user's request hanging for 5 seconds instead of 2 seconds due to retries isn't just a token cost — it's potential abandonment, and each 100ms of latency costs you revenue. If your retry strategy is adding 3+ seconds to p99, you might be better off failing fast and letting the user retry from their side.

---

Continue reading: the rest of this post — including the sections that follow — is on Loop & Retry: read the full article