You call an LLM API and it times out. So you retry.
The retry gets a 429. You wait and try again. This time the provider responds, but the output fails validation. Maybe you have a fallback provider, so you call that instead.
Now the interesting question is no longer whether you know how to retry an HTTP request.
It is how all of those decisions interact.
This problem shows up constantly in AI systems because model APIs are external, relatively expensive and sometimes unpredictable. Providers have different rate limits, latency profiles and failure modes. A request may also have side effects elsewhere in your system, so blindly repeating it is not always safe.
Retries, timeouts, fallbacks, circuit breakers and rate limits are individually simple concepts. The difficult part is defining what should happen when several of them apply to the same operation.
Should validation failure trigger a retry?
Should a fallback run after a timeout?
What happens if the fallback fails too?
If a provider tells you to retry after 20 seconds but your normal backoff says 2 seconds, which one wins?
If the operation eventually succeeds after four attempts, do you report success or the full sequence of failures that led to it?
Those questions are what led me to build RelPrim.
RelPrim is a Python library for reliability around external operations. It models retries, timeouts, fallbacks, circuit breakers, idempotency, rate limit recovery and validation together rather than treating each one as an unrelated wrapper.
The same model works for LLM providers, payment APIs, internal services or any other external dependency.
It also keeps structured execution information about what happened. If an operation succeeds after several retries or through a fallback, you can inspect that instead of seeing only the final return value.
I have just released RelPrim 1.0.
Most of the work before the stable release was not adding new features. It was tightening cancellation behavior, typing, policy interactions and the public API so that combining these mechanisms stays predictable.
The project is here:
https://github.com/bart-rozycki/relprim
Documentation:
https://bartrozycki.com/relprim
I would be interested in hearing how people building AI systems handle reliability across multiple model providers, especially when retries, rate limits, validation and fallbacks start interacting.
Top comments (0)