The Retry Loop That Outran Your Health Check

You closed the laptop after a quiet Friday deploy, convinced the status page would stay green until morning. Two hours later the health check had turned red, even though the process was still running and burning time. The log was not a crash dump, but a long trail of polite retries that never learned how to wait. One downstream timeout had become your outage, mostly because nothing in the diff looked dangerous enough to reject.

This walkthrough is a reconstructed incident, not a claim that one named dashboard failed on a dated night. You can rebuild the same timeline from ordinary logs whenever a side project starts consuming its own free capacity. Run the commands on a laptop first, and treat every snippet as a lab procedure you execute yourself. Nothing here is a measured production report, and no latency or quota figure is being offered as evidence.

What the clock actually showed

The first useful minute is a clock, a process list, and the last health response you can still fetch locally. On a small Linux host that sequence is short, and it separates a wedged app from one that is merely loud. You are looking for a live process that misses a short timeout, which is a different failure from a dead process. If the process is gone, this postmortem is the wrong map, and you should start with the supervisor instead.

date -u +"%Y-%m-%dT%H:%M:%SZ"
ps -eo pid,etime,pcpu,pmem,cmd | awk 'NR==1 || /node|python|uvicorn/'
curl -sS -o /tmp/health.body -w "http=%{http_code} time=%{time_total}\n" \
  --max-time 3 http://127.0.0.1:8080/health || true
tail -n 80 /var/log/app/app.log

In the reconstructed timeline, the deploy finishes at 21:14 UTC, the health route returns 200, and the process sits near idle. At 21:41 a downstream status feed times out once, and the handler schedules another attempt with no delay and no cap. By 22:06 the log shows a crowd of attempts against the same dead feed, and the health route starts missing its budget. You leave the process running until 22:31 because it still looks alive, even though the feed itself recovered around 22:09.

Three factors usually stack into this shape, and none of them looks fatal when you inspect it alone. The draft handler optimized for not dropping the request, which is a reasonable local goal and a poor global one. The host gave you a place to run, but it did not give you infinite processor time or a person waiting on call. Your health check only proved that the process existed, so a busy loop still looked fine until the timeout fired.

A smarter drafting model does not retire this class of bug, even when the surrounding tools feel newer than last season. The model is rewarded for completing the unhappy path, while your users are punished when that path never yields the processor. Software still fails at the seams between a helpful suggestion and a shared budget, which is an old operations fact wearing a new interface. Treat the generated handler as an untrusted patch, and make the budget the part you refuse to delegate.

When you write the note afterward, keep four timestamps and one sentence of cause, because a long narrative rots faster than a log line. Record when the bad attempt started, when health first missed, when the dependency recovered, and when you actually stopped the process. The gap between recovery and your stop is the part you own, and a later review should still find that number. If that gap is missing, the document is a recap, and a recap will not change the next Friday deploy.

Disclosure: This article was prepared as part of MonkeyCode's product outreach. Free model access fits as a drafting partner for the retry budget, and a free server fits as the rehearsal stage. Those two availability claims are the only product facts this walkthrough relies on, and both should be rechecked in current docs. This article does not name models, token quotas, hardware sizes, or how long either offer remains available.

The budget that should have existed

The durable fix is not a slogan about retrying less, but a budget you can read and fail in a test. A retry may run only inside a deadline, only with jittered backoff, and only for the error you explicitly named. Anything else fails closed, records the miss, and leaves the health route free to answer from the last known state. That rule is the incident compressed into one policy, and the code below is how you keep the policy from drifting.

import random
import time
from dataclasses import dataclass

@dataclass(frozen=True)
class RetryBudget:
    max_attempts: int = 3
    base_delay_s: float = 0.2
    deadline_s: float = 2.0

def call_with_budget(op, budget: RetryBudget, sleep=time.sleep, now=time.monotonic):
    started = now()
    last_error = None
    for attempt in range(1, budget.max_attempts + 1):
        if now() - started >= budget.deadline_s:
            break
        try:
            return op()
        except TimeoutError as exc:
            last_error = exc
            if attempt == budget.max_attempts:
                break
            delay = budget.base_delay_s * (2 ** (attempt - 1))
            delay += random.uniform(0, budget.base_delay_s)
            remaining = budget.deadline_s - (now() - started)
            if remaining <= 0:
                break
            sleep(min(delay, remaining))
    raise TimeoutError("retry budget exhausted") from last_error

The handler above is proposal code until you run it, so do not treat the listing as a result from a live host. It caps attempts, adds a little jitter, and refuses to sleep past the deadline you already promised the caller. You can replace sleep and the clock in tests, which is what makes the budget provable without a real network. If a later edit removes either guard, the next test should fail before that change reaches any server.

def test_retry_stops_at_budget():
    calls = {"n": 0}
    clock = {"t": 0.0}

    def op():
        calls["n"] += 1
        raise TimeoutError("feed down")

    def sleep(seconds):
        clock["t"] += seconds

    def now():
        return clock["t"]

    try:
        call_with_budget(op, RetryBudget(), sleep=sleep, now=now)
        raised = False
    except TimeoutError:
        raised = True
    assert raised
    assert calls["n"] <= 3
    assert clock["t"] <= 2.0

The test stubs both sleep and the clock, so a red result means the policy broke, not that some feed was slow that day. Paste both snippets into files, run the pytest command below, and only then debate whether three attempts feels generous. If the call count exceeds three, the loop is unbounded again, and the Friday failure mode has quietly returned. If the fake clock passes two seconds, backoff outgrew the deadline, and health will time out while the feed is merely slow.

A free server is the right stage for this rehearsal, and the wrong place to meet the same loop through real users. Start the app, point the feed at a port that refuses connections, and watch whether health stays inside one second. Replace the paths with your own, because these commands are a shape, not a claim about any particular host image. Read the five samples as a short story: every health line should return, even while the feed stays dead on purpose.

python -m pytest -q test_retry_budget.py
FEED_URL=http://127.0.0.1:9/status python app.py &
APP_PID=$!
for i in 1 2 3 4 5; do
  curl -sS -o /dev/null -w "%{http_code} %{time_total}\n" \
    --max-time 1 http://127.0.0.1:8080/health || echo "health missed"
  sleep 1
done
kill "$APP_PID"

If a sample prints that health was missed, the worker still shares a loop with the route that users actually call. Move the feed poll onto a background task that owns the same budget, and keep health as a pure read of last state. A downstream timeout then becomes a stale badge, which is an honest degraded mode rather than a self-made outage. That split is the durable fix, and the test is what stops the next helpful edit from merging the two paths again.

Who should leave this lab alone

Skip this approach when the service pages a human, takes payment, or stores data you cannot afford to lose. Do not let a free model write retry, authentication, or migration code that you cannot test on the same day. Do not read this postmortem as evidence about any vendor uptime, because no latency table was collected and no quota was measured. If you already keep an incident process, keep it, and use this lab only as a drill before the next deploy.

A retry without a timer is someone knocking on a closed door until persistence starts to look like damage. A budget is the kitchen timer you set before the first knock, so the rest of the house stays usable when it rings. A generated draft can sketch that timer, and a spare host can stage the rehearsal, but neither owns the assertion. If you use MonkeyCode's free model access and free server for that drill, read the current docs and keep the failing test in git.