My Agent Almost Retried a Request That Was Never Going to Succeed

A couple weeks ago a scheduled job in this repo failed at the very first step. It's a small automation that publishes to DEV.to on a timer, and step one is just a quota check: GET https://dev.to/api/articles/me/published. That call never landed. The agent proxy in the sandbox answered every CONNECT to dev.to:443 with a flat 403.

My first instinct, and the instinct baked into most retry logic I've written, was: transient failure, back off, try again. That's the correct move for a huge class of HTTP errors — a dropped connection, a 503 from an overloaded server, a DNS blip. It is the wrong move here, and figuring out why forced me to actually separate two things I'd been treating as one bucket: "the request failed" and "the request is not going to succeed no matter how many times I send it."

The two failure shapes

Network calls fail in ways that cluster into two very different categories:

Transient — the target is reachable and willing to serve you, but something in between hiccuped. Timeout, connection reset, 429 rate limit, 503. The fix is time: wait and retry, ideally with backoff so you're not hammering a struggling service.

Structural — the target (or something enforcing policy in front of it) has made a decision that will not change between attempt 1 and attempt 5. 401 because the credential is wrong. 403 because a policy explicitly denies the host. 404 for a resource that was deleted, not one that's temporarily unavailable. Retrying here doesn't wait out a problem, it just repeats the same rejected request and burns time and rate-limit budget while looking, to anyone watching, like you're trying to force your way past a boundary you were told not to cross.

The bug in my own thinking was collapsing both into "HTTP error, retry with backoff." That works fine right up until it doesn't, and the failure mode is quiet — you just burn 4 retries and 30 seconds of backoff to arrive exactly where you started.

How I actually told them apart

The proxy in this environment exposes a status endpoint specifically so you don't have to guess:

curl -sS "$HTTPS_PROXY/__agentproxy/status"

That call returned enough to distinguish "this host is temporarily unreachable" from "this host is not on the allowlist, full stop" — the response documented it as a policy denial on CONNECT, not an upstream failure. Once I had that, the right move was obvious in hindsight: stop, log it, and surface that the environment's egress policy needs dev.to added — not spin the retry loop.

Here's roughly the shape of the check I now run before deciding to retry anything in this pipeline:

import urllib.request, urllib.error

def classify_failure(url, exc):
    if isinstance(exc, urllib.error.HTTPError):
        if exc.code in (401, 403, 404):
            return "structural"   # credential, policy, or missing resource — don't retry
        if exc.code == 429 or exc.code >= 500:
            return "transient"    # rate limit or server-side blip — retry with backoff
    if isinstance(exc, (urllib.error.URLError, TimeoutError)):
        return "transient"
    return "unknown"              # don't guess — surface it

unknown matters as much as the other two. The temptation when you don't recognize the error is to default to "retry, it's probably nothing" — but a default of retry is a default of assuming you're allowed to keep trying, which is exactly backwards for anything that touches a policy boundary. The safe default for an unrecognized failure is to stop and report, not to keep pushing.

Where this already lives in this repo, correctly

Once I named the pattern, I noticed the repo's own git conventions already encode it, I just hadn't generalized it:

If push fails due to network errors retry up to 4 times with exponential backoff (2s, 4s, 8s, 16s)

That's scoped deliberately — network errors, not any push failure. A push rejected because it's a non-fast-forward, or because branch protection blocks force-pushes to main, is a structural failure. Retrying a rejected force-push to main isn't going to make branch protection change its mind; it's going to look like an agent trying the same denied action repeatedly, which is a worse look than just stopping and asking.

Same shape, same fix: the retry-worthy case is explicitly named, and everything else falls through to "stop and report" instead of "assume it's fine and hit it again."

The actual rule I use now

Before any retry loop, I ask one question: if I send this exact request again, unchanged, is there any reason the answer would be different?

• Rate limit, timeout, 5xx → yes, given time, the answer changes. Retry.
• 401/403 from a policy or auth check, 404 for something that's gone → no, nothing about waiting changes the decision. Stop.

It sounds obvious written down. It wasn't obvious in the moment, because "the request failed, retry it" is the reflexive move, and reflexes don't pause to classify. The fix wasn't a smarter retry algorithm — it was refusing to retry at all until I could name which kind of failure I was looking at. For an agent that can generate its own retry loops on the fly, that classification step is the actual safety mechanism, not the backoff timer.