Retries Don't Fix a Flaky Dependency: Five Beliefs and an Amplification Probe

How many requests did your client actually send last night?

Most teams can't answer that. They know how many logical calls they made. They don't know the wire count. That gap is where retries turn a small hiccup into a self-inflicted outage.

I don't want to invent an outage story here. So instead of anecdotes, this post gives you a probe. You run it, you read two numbers, and you stop arguing about retry policy in code review.

The five beliefs I keep hearing

1. "A retry is basically free"

It isn't. Every attempt is a new request on the wire, a new queue entry, and usually a new token charge. When a dependency degrades, retries are the thing that keeps the queue full. Ask yourself: if all callers retry three times, what does the dependency see? It sees 3x traffic at the exact moment it has the least capacity.

2. "We have exponential backoff, so we're safe"

Backoff alone synchronizes clients. Everyone waits the same 0.5s, then the same 1s, then hits you again together. AWS's Builders' Library covers this well: you want jitter, and full jitter is the simple version. Backoff without jitter just moves the spike later.

3. "Any 5xx means try again"

Not quite, and not only 5xx. RFC 9110 tells you which methods are idempotent, and that's the property your retry loop depends on. A 503 on a GET is a different animal from a 503 on a POST that created a row. And a 400 will fail identically forever. Retrying it is pure amplification.

4. "Idempotency keys are a nice-to-have"

They're the whole ballgame for writes. If your retry can duplicate a side effect, you don't have a retry policy. You have a duplicate policy waiting to be discovered by finance. One stable key per logical call, reused across attempts, is the minimum.

5. "More attempts equals more resilience"

Attempt count is not your safety net. The deadline is. A caller with 800ms left cannot afford attempt four. Cap by wall-clock budget first, attempts second. Otherwise your retries outlive the request that needed them.

Build the artifact: a fault-injecting server

You need failures on demand. Here's a tiny scripted server. It's stdlib only, so no install step.

# flaky_server.py — stdlib only, Python 3.11+
import itertools
from http.server import BaseHTTPRequestHandler, HTTPServer

SCRIPT = [503, 503, 200]  # one cycle = one logical call
total = 0

class Handler(BaseHTTPRequestHandler):
    seq = itertools.cycle(SCRIPT)

    def do_POST(self):
        global total
        length = int(self.headers.get("Content-Length", 0))
        self.rfile.read(length)
        total += 1
        code = next(Handler.seq)
        self.send_response(code)
        self.send_header("Content-Type", "application/json")
        if code == 429:
            self.send_header("Retry-After", "1")
        self.end_headers()
        self.wfile.write(b'{"ok": %s}' % (b"true" if code == 200 else b"false"))
        if code == 200:
            print(f"wire requests so far: {total}", flush=True)

    def log_message(self, *args):
        pass

HTTPServer(("127.0.0.1", 8099), Handler).serve_forever()

Start it, then run the probe against it. Swap SCRIPT for [429, 200] or [500] * 5 and watch the numbers move.

Build the artifact: the amplification probe

# probe.py — stdlib only, Python 3.11+
import json, random, time, urllib.error, urllib.request, uuid

RETRYABLE = {408, 425, 429, 500, 502, 503, 504}

def call(url, payload, key, timeout=5.0):
    req = urllib.request.Request(url, data=json.dumps(payload).encode(), method="POST")
    req.add_header("Content-Type", "application/json")
    req.add_header("Idempotency-Key", key)  # stable across attempts
    start = time.perf_counter()
    try:
        with urllib.request.urlopen(req, timeout=timeout) as r:
            return r.status, time.perf_counter() - start
    except urllib.error.HTTPError as e:
        return e.code, time.perf_counter() - start
    except (urllib.error.URLError, TimeoutError):
        return None, time.perf_counter() - start  # unknown state

def retrying_call(url, payload, attempts=5, budget_s=20.0, base=0.5, cap=8.0):
    key = str(uuid.uuid4())  # one logical call, one key
    deadline = time.monotonic() + budget_s
    log = []
    for i in range(1, attempts + 1):
        status, seconds = call(url, payload, key)
        log.append({"attempt": i, "status": status, "seconds": round(seconds, 3)})
        if status is not None and 200 <= status < 300:
            break
        if status is not None and status not in RETRYABLE:
            break  # deterministic failure, stop burning budget
        if i == attempts:
            break
        delay = random.uniform(0, min(cap, base * 2 ** (i - 1)))  # full jitter
        if time.monotonic() + delay > deadline:
            break
        time.sleep(delay)
    return log

if __name__ == "__main__":
    log = retrying_call("http://127.0.0.1:8099/v1/thing", {"n": 1})
    for row in log:
        print(row)
    wire = len(log)
    print(json.dumps({
        "logical_calls": 1,
        "wire_requests": wire,
        "amplification": round(wire / 1, 2),
        "spent_s": round(sum(r["seconds"] for r in log), 3),
        "terminal_status": log[-1]["status"],
    }))

Run it against [503, 503, 200] and you should see three wire requests and amplification: 3.0. Run it against a healthy dependency and you should see 1.0. That baseline comparison is the whole point.

How to read the two numbers

• Amplification — wire requests divided by logical calls. Healthy is 1.00. Anything above it is load you added during a bad minute.
• Spent seconds — sum of attempt latency, not wall clock. If it exceeds the caller's budget, your retry loop is lying about time.
• Terminal status — what the caller finally saw. A None here means unknown state, not failure.
• Client vs server counts — compare against the server's own counter. A mismatch means a retry layer you don't control (SDK, proxy, service mesh).

That last bullet catches most surprises. Many HTTP clients retry silently. If your SDK retries and your loop retries, your real amplification is the product, not the sum.

Where a free host actually helps

The probe is more honest when client and server are on different machines. Timeouts, NAT behavior, and clock skew all change the numbers on the same box. MonkeyCode's free server option is enough for that split, and the free model access gives you a real POST endpoint to point the probe at without buying a key first. Disclosure: This article was prepared as part of MonkeyCode's product outreach. Keep the fault-injecting server local; point only the client side at a remote host. You are measuring your retry loop, not the provider.

A four-row decision table

Observed	Retry?	Reason
429 with Retry-After	Yes, after that delay	The server told you when
500/502/503/504	Yes, jittered, deadline-capped	Assume transient, assume shared
400/401/403/404/422	No	Deterministic, retry is pure waste
Timeout on a POST without a key	No	Unknown state, you may duplicate

Limitations, and who should skip this

This probe is one client against one scripted server. It doesn't model queueing depth or server-side shedding. It doesn't validate your jitter distribution statistically. And a 200 still doesn't prove the work was correct — you need a dedupe check on the server side for that.

Skip this approach if any of these are true:

1. Your writes have side effects and you have no idempotency key. Fix that first; retries are unsafe until then.
2. You can't disable SDK-internal retries. You'll measure a number you can't attribute.
3. Your endpoint is streaming or stateful. Replayed partial state is a different failure class.
4. You need a load test. This is a correctness probe, not a capacity test.

The question to ask in review

Not "do we retry?" Everyone retries. The question is: "what's our wire count when the dependency is half dead?" If nobody in the room can answer, run the probe. It takes about ten minutes, and the table above is shorter than the argument.

If you want the client on a second host, the free server option is the cheapest way I know to get that separation.