DEV Community: Emre Kadir Dağdelen

Make your LLM API calls resilient in Python

Emre Kadir Dağdelen — Thu, 23 Jul 2026 07:45:05 +0000

If your app calls an LLM API (OpenAI, Anthropic, anything), you have already
met its failure modes: 429 rate limits, occasional 500s, requests that hang,
and the tail latency where one call in fifty takes ten seconds. In a demo you
ignore this. In production it is most of your incidents.

Here is a compact way to handle all of it, using nopanic (pip install
nopanic), a zero-dependency resilience toolkit. Everything below works the
same on sync and async functions.

The problem, honestly

A naive call looks like this:

async def ask(prompt):
    return await client.chat.completions.create(...)

Every failure mode above takes down the request. Worse, the naive fix
(wrap it in a retry loop) can make an outage worse: if the provider is
struggling and everyone retries immediately, the retries pile on.

Retry, but politely

from nopanic import retry, backoff

@retry(attempts=4, on=RateLimitError,
       backoff=backoff.full_jitter(base=0.5, cap=30.0))
async def ask(prompt): ...

Full jitter spreads the retries out randomly so they do not arrive in a
synchronized wave. If the error carries a Retry-After hint, nopanic honors
it (capped, so a hostile value cannot park your client).

Stop hammering a provider that is down

from nopanic import circuit_breaker

llm = circuit_breaker(failure_threshold=0.5, reset_timeout=20.0)

@retry(...)
@llm
async def ask(prompt): ...

Once half the calls in the window fail, the breaker opens and fails fast for
20 seconds instead of sending doomed requests. Then it lets one probe through
to check if the provider recovered.

Bound every attempt, and degrade instead of crashing

from nopanic import compose, fallback, timeout, CircuitOpen

resilient = compose(
    fallback(lambda e: "Sorry, please try again shortly.",
             on=(RateLimitError, CircuitOpen, TimeoutError)),
    retry(attempts=3, on=RateLimitError, backoff=backoff.full_jitter(0.5)),
    llm,
    timeout(30.0),
)

@resilient
async def ask(prompt): ...

Read it from the inside out: each attempt gets 30 seconds, outcomes feed the
breaker, rate limits retry with jitter, and anything still unhandled becomes
a graceful message instead of a stack trace reaching your user.

Fix the tail, not just the median

For idempotent reads, if a call is slow, race a second one:

from nopanic import hedge

@hedge(delay=0.8)
async def embed(text): ...

If the first attempt has not answered in 800 ms, a duplicate is sent and
whichever returns first wins. This trades a little extra load for a much
better p99.

The point

None of these patterns are new. What is annoying is wiring them together from
three different libraries. Putting the whole stack in one place, with one API
that reads top to bottom, is the entire idea. Full docs and source:
github.com/dagdelenemre/nopanic

A circuit breaker bug that turned 3.2M calls into 40 minutes

Emre Kadir Dağdelen — Wed, 22 Jul 2026 12:18:42 +0000

I maintain nopanic, a small resilience toolkit for Python (retries, circuit
breakers, timeouts, rate limits). Everything passed its tests, so I did what
I should have done earlier: I put it under real load. The result taught me a
lesson worth writing down.

The setup

A circuit breaker watches the failure rate of a dependency. If too many
calls fail inside a time window, it "opens" and fails fast instead of
hammering something that is already down. To know the failure rate, it has
to remember recent outcomes.

My first implementation remembered them the obvious way: a list of
(timestamp, ok/fail) records inside a sliding time window. On every failure
it counted how many entries in that list were failures, divided by the
total, and compared to the threshold.

Correct. All tests green. Shipped.

The load test

I wrote a stress test: 32 threads, 100,000 calls each, one shared breaker,
about a third of the calls failing (below the trip threshold, so the breaker
stays closed and keeps recording). 3.2 million calls total.

It took 2,413 seconds. Forty minutes. Single threaded, the breaker does
roughly 850,000 calls per second, so those calls should have finished in
about four seconds.

The diagnosis

Two problems, one root cause.

First, memory. The window kept one record per call. At high throughput that
list holds every call inside the window. I measured it: one million calls in
the window was about 88 MB, for a single breaker. It grows with traffic,
without bound.

Second, and worse, time. On every failure the code recomputed the failure
count by scanning the entire list. As the list grew, each scan grew with it.
N failures, each doing O(N) work, is O(N squared). Under sustained partial
failure the whole thing collapses into a quadratic crawl, and because the
scan happens inside the breaker's lock, all the threads pile up behind it.

That is the forty minutes.

The realization

The fix was not a faster scan. It was noticing that a circuit breaker's
memory is not a log. It only needs to answer one question: what is the
failure rate right now. It never needs the individual records. I was storing
a detailed history to compute a single ratio.

Detailed per-request history is a real need, but it belongs in an
observability stream that the user forwards to their own logging, not in the
breaker's hot path. Keeping millions of records in RAM to compute one
percentage was the actual bug.

The fix

Replace the per-call list with a fixed set of time buckets, each holding two
integers: how many calls, how many failures. Keep running totals updated as
buckets rotate out of the window. This is how Hystrix and resilience4j do
it, and now I understand why.

Recording an outcome: O(1).
Reading the failure rate: O(1), from the running totals.
Memory: constant, no matter the traffic. Ten buckets of two integers.

The same 3.2 million call load now finishes in about 2.4 seconds instead of
40 minutes. One million in-window calls use 0.001 MB instead of 88. The
per-call cost on the success path even dropped slightly.

The trade-off: outcomes now expire in bucket-sized steps instead of at an
exact per-call age. For deciding whether to trip a breaker, that precision
was never worth its price.

Two lessons

First, tests that check correctness do not check behavior under load. My
suite was green the entire time the library was quadratic. A partially
failing dependency at high traffic, the exact situation a circuit breaker
exists for, was the situation that broke it.

Second, a bonus one I tripped over while fixing this: I was testing with an
editable install, and a stale copy of the package in site-packages was
shadowing my source. My tests were passing against old code. Now the CI also
builds the wheel, installs it into a clean environment, and runs the suite
against the installed artifact. Never fully trust pip install -e alone.

The library is nopanic (pip install nopanic) if you want to see the code

Building the Polly that Python never had

Emre Kadir Dağdelen — Mon, 13 Jul 2026 10:53:15 +0000

Building the Polly that Python never had

Every app that calls external APIs faces the same four problems: transient
failures, dead dependencies, rate limits, and slow responses. Java solved
this with resilience4j. .NET solved it with Polly. Python never had a
unified answer: tenacity does retries, pybreaker does circuit breaking
(sync only, aging), and everything else you wire by hand.

These patterns only pay off when they compose: a timeout per attempt,
retries that respect an open circuit, a fallback that absorbs the rest.
Wiring that from three libraries with three philosophies is the code
nobody wants to own. So I built nopanic.

One decorator API, sync and async

from nopanic import compose, fallback, retry, circuit_breaker, timeout, backoff, CircuitOpen

llm = circuit_breaker(failure_threshold=0.5, reset_timeout=20.0, name="llm")

resilient = compose(
    fallback(lambda e: "temporarily unavailable", on=(ConnectionError, CircuitOpen, TimeoutError)),
    retry(attempts=3, on=(ConnectionError, TimeoutError), backoff=backoff.full_jitter(0.2)),
    llm,
    timeout(30.0),
)

@resilient
async def ask(prompt: str) -> str: ...

Reading inside out: each attempt gets 30 seconds, outcomes feed the
breaker, transient failures retry with jitter, and whatever is left
becomes a degraded answer instead of a traceback. The same decorators
work on plain sync functions. Zero runtime dependencies.

Design choices that mattered

No wrapper exceptions. Exhausted retries re-raise the original error.
Your except ConnectionError: keeps working.

Injectable time. Breakers, rate limiters and caches take a clock=
parameter, so the test suite (141 tests) runs in under two seconds with
zero sleeps.

Observability as a stream. events.subscribe() sees everything every
policy does. With no listeners the cost is one attribute read (~0.1us).

Measured overhead. Every policy costs 0.15 to 1.3 microseconds per
call on the success path. A fast HTTP round trip is 5,000+ us.

Validation: retrofitting a 232k-star repo

To test the API against reality, I took network scripts from
geekcomputers/Python and added resilience: three decorators per call
site, no restructuring. The exercise even improved the design: my first
retry policy naively retried a 403, which led to the documented
"retry 429/5xx, never other 4xx" recipe.

The bug that only exists on Windows

The best story came from CI. Tests passed everywhere except Windows on
Python 3.11/3.12. After reproducing locally and instrumenting, the data
showed time.sleep(0.05) waking after 47ms by the monotonic clock, and
sub-quantum sleeps returning instantly. My retry honored a server's
Retry-After by sleeping exactly that long, then racing the deadline and
losing, burning attempts against a still-closed breaker window.

The fix belongs in the library, not the test: "retry after X" means "not
before X", so honored hints now sleep X * 1.05 + 50ms, still capped so a
hostile server cannot park your client. If your code races deadlines
exactly, Windows will eventually teach you the same lesson.

Adaptive rate limiting: stop hardcoding guesses

The newest addition applies TCP's AIMD congestion control to API quotas:
a 429 cuts your rate multiplicatively, successes recover it additively,
and Retry-After blocks the bucket for exactly that long. You converge on
the rate the server actually sustains.

pip install nopanic — GitHub: github.com/dagdelenemre/nopanic
Feedback and issues welcome, especially from Polly and resilience4j users.