Originally published on hexisteme notes.
For about three months, I had a working theory about one leg of my model roster: the free tier was flaky. Every so often, a request routed through an NVIDIA NIM model would come back 503. My client would retry on a one-second backoff, then a three-second backoff — retry_transient: [502, 503, 504] — and both would also fail. Then, roughly a minute later, with no intervention from me, the same request would go through fine. I filed it under "free tier, what do you expect," because a failure that heals itself on its own timetable is what a rate-limited endpoint is supposed to look like.
The theory survived because the evidence fit it every time. What broke it wasn't a new outage — it was a probe result and one column in an access log I'd been reading past for months. The vendor wasn't the unreliable part. A gateway I run myself was — and, as it turned out, just one of four separate causes behind the unstable leg I was chasing that same day, the only one that actually lived inside the gateway.
The column I wasn't reading
The gateway is CLIProxyAPI 7.1.58, a multi-credential proxy — the kind of thing grouped with LiteLLM-class routers. It runs locally; everything downstream of it, including an MCP tool server's custom provider and a council persona whose OPENAI_BASE_URL points at it, reaches it on a local port. Its access log writes one line per request, with a latency column I'd been skimming past:
[gin_logger.go:97] 503 | 6ms | 127.0.0.1 | POST /v1/chat/completions
Once I actually read that number across a run of failures, the pattern was obvious. The first 503 in an episode carried a latency of 316 ms — a real round trip, long enough to actually reach NVIDIA and come back with a failure. Every 503 that followed in the same episode came back in 6 ms, and in older episodes as fast as 1–2 ms — the gateway answering from memory before the request ever left the building. A direct probe to the vendor, at the same moment one of these episodes was running, came back 200. A status code alone doesn't say which layer failed — the same discipline applies in reverse, where an error envelope inside an HTTP 200 still reads as success if the only check is response.ok (An Error Inside HTTP 200 Poisoned My Cache). Here, a 503 looked exactly like a vendor failure and wasn't one; the only way to tell them apart was to time it.
What a cooldown is for, and what it becomes at one key
A gateway holding more than one credential for a provider uses a cooldown to protect itself: if key A errors, set it aside and route the next request through key B. In CLIProxyAPI's code, that logic lives in sdk/cliproxy/auth/conductor.go — case 408, 500, 502, 503, 504: sets NextRetryAfter = now.Add(1 * time.Minute) unless disable-cooling is set on that provider block (429 runs through a separate quota path). The per-provider switch lives at internal/config/config.go:658.
That logic assumes a key B. My NVIDIA NIM provider block had exactly one API key configured — one api-key-entries entry. So the same cooldown that's a sensible rotation with two keys instead put the entire provider to sleep for that minute: zero available auth, for the only credential it had. For the rest of that window the gateway answered every request itself, immediately, with 503 auth_unavailable / "no auth available," without ever contacting NVIDIA. That's the 6 ms.
The arithmetic downstream was already decided before the cooldown even started. My retry policy backs off at 1 s and then 3 s, both well inside the 60-second window the cooldown holds open — not because the vendor failed again, but because a minute-long timer doesn't care how fast a client retries. The seed for a given episode was ordinary on its own — NVIDIA NIM's free tier returning "Service temporarily overloaded," or once a 302-second hang ending in a 504 — but the blackout that followed had nothing more to do with how bad that seed event was.
Every fix I almost made
- Raise the consumer's retry count or timeout — the blackout is fixed at 60 seconds regardless of cause; three retries or five, all land inside the same window.
- Blame the free tier again — the direct probe answered
200at the same moment the gateway refused the request. The outage never left my machine. - Restart the gateway to clear the cooldown — it works, because the cooldown is in-memory state, which is the problem: the restart clears the symptom and leaves the cause in place, and cuts every other consumer on the gateway for a fix only one leg needed.
- Turn
disable-coolingon globally — wrong for any provider that actually holds two or more keys, where the cooldown is doing its real job. The setting belongs per provider, sized to how many keys that provider has. - Treat every non-2xx the same way. A
410means a model has genuinely reached end of life — permanent, needs its alias redirected, not retried. A429runs through the gateway's own quota path. Folding either into "the leg is flaky" erases what tells you what to do next.
The fix
For a provider with one key — the default case — the fix is disable-cooling: true on that provider's block only:
openai-compatibility:
- name: "nvidia-nim"
base-url: "https://integrate.api.nvidia.com/v1"
disable-cooling: true
api-key-entries:
- api-key: "<redacted — injected from env>"
CLIProxyAPI hot-reloads its config file via a watcher — the log records "config successfully reloaded" — no restart, no gap for other consumers. I applied this at 2026-09-05 20:41:58 and reran the probe: PASS 3/3 — nano in 3972 ms, super in 558 ms, and the 49b model correctly back with 410, its expected end-of-life answer, not a new failure.
For a provider where I can hold two or more keys, the fix runs the other way: add another API key, let the cooldown become a real rotation, and leave it on. Custody matters here for the same reason it mattered when a metered API key, exported globally in a shell profile, turned out to be inherited by unrelated tools that had nothing to do with the one originally suspected (The Key's Owner Wasn't the Spender) — each key gets injected per consumer from the environment, never printed in the clear.
Either way, retries get exactly one owner. Mine live in the consumer's roster — retry_transient plus its backoff — and nowhere else. Stack a retry loop on top of a gateway-side cooldown and you have two layers with two different budgets deciding the same failure — and here the smaller budget was always going to lose.
The diagnostic itself is just the latency column, grepped and tailed:
grep -E "gin_logger.go:[0-9]+\] 503 " ~/path/to/gateway/launchd.out | awk -F'|' '{print $1 "|" $2}' | tail -20
Run against the history, the same signature — one slow 503, then a run of 1–2 ms 503s — showed up on 2026-06-25, 2026-07-01, and 2026-08-26, and again on 2026-08-27 and 2026-09-02, the two episodes I'd gone looking for. The access log held 37 lines like that; most of them carried the signature. It had been there the whole time I was blaming the free tier.
A different layer, same misattribution
This sits in the same family as a tail -20 that couldn't emit until EOF, with the pipe's write-end held open by an inherited descriptor — the tool was fine there too, and an observation pipeline manufactured the failure (For Weeks I Logged a CLI as Flaky). The roles are flipped here: the vendor was fine, and my own gateway layer manufactured the outage. Same instinct to blame the far end of the wire first; the defect was one layer closer to home both times.
What generalizes
- A protective mechanism built for N ≥ 2 of something — keys, replicas, nodes — can become the failure it exists to prevent once N = 1. The safety property was never in the code path; it was an assumption about the environment around it that nothing enforces.
- A status code from a proxy isn't automatically the vendor's status code. Time the request before deciding where it failed: milliseconds mean the network was never touched; hundreds of milliseconds to seconds mean somebody upstream answered.
- Retries and cooldowns are both timers with an opinion about failure. When a client's retry budget is smaller than a server-side cooldown window, every retry is a foregone conclusion — the fix is one layer owning retries, not a bigger timer.
- When several things break on the same day, check whether they even share a layer before treating them as one root cause. Symptoms carrying the same "unstable" label can still have separate homes.
Where this stops being true
- With two or more keys behind a provider, the cooldown is doing its actual job. Disabling it there removes a legitimate rotation, not a false one.
- If the
503's latency in the log reads in the hundreds of milliseconds or seconds, that request went upstream and came back with a real failure — a vendor problem again, not a local blackout. - If a gateway ever starts skipping cooldown by default when a provider has exactly one credential, or shrinks the cooldown window below a consumer's retry budget — a few seconds instead of a minute — this fix stops being necessary.
None of the individual facts here were hidden. The code was right there to read, the log line carried a latency column the whole time, and the vendor answered honestly when I asked it directly. The signature sat in that log for about three months before I read it as a local blackout instead of vendor instability — not for lack of evidence, but because I hadn't yet thought to distrust the layer I'd installed to make the fleet steadier in the first place.
Email list for these notes: hexisteme.beehiiv.com — no issue has gone out yet, so you would be on it before the first one. No welcome sequence, no course, no upsell.
More notes at hexisteme.github.io/notes.
Top comments (2)
The one-key case is what gets me about circuit breakers and cooldowns generally. With two keys, a single 503 drops you to half capacity for 60 seconds and routing continues. With one key, the same event turns the cooldown from a rotation into an on/off switch: full availability to zero, for the entire window. I've seen the same shape with connection pools configured for failover that had no secondary to fail to.
One caveat on "two or more keys means the cooldown is doing its job": that holds for a per-key quota 503, not an upstream-capacity one — with "Service temporarily overloaded" as the seed, key B just buys another round trip to the same broken endpoint, so the fix there is failing over to another provider.
And since auth_unavailable is the one 503 that never left your machine, it's worth returning Retry-After on it (so a client's budget can align with the window instead of burning inside it) and alerting on it as a counter rather than grepping for it afterwards.