I'm building Motiontator, an OpenAI-compatible gateway with one endpoint and one balance across multiple model providers. One design problem has taken more care than the routing itself: deciding what to do after a request times out.
I also handle supplier review manually, including investigating suspected model substitution, undisclosed downgrades and unusual output. Those checks have limits. This article focuses on the request and billing behavior around provider failures.
A gateway can stop receiving data while the provider keeps working. That leaves two separate questions: did the user receive a complete answer, and did the upstream do work that may be billed?
Treating both questions as one success flag makes retry logic and accounting harder to reason about.
A small failure timeline
Consider this hypothetical sequence, rather than a customer incident:
- The gateway submits a request to provider A.
- A accepts it, but the connection times out before the gateway receives a terminal result.
- The gateway submits the same prompt to provider B.
- B returns a complete answer with usage data.
The user received an answer. B has a known cost. A's execution and cost remain unknown until there is enough evidence to resolve them. B succeeding doesn't establish that A did no work.
This matters even more when an application can call tools. A repeated model request may produce another tool call, and the application could execute that action twice. Payment operations, emails and data writes need their own safeguards.
Keep request outcome and billing state separate
I use separate states for the answer delivered to the application and the money involved in producing it.
The application-facing outcome can be completed, failed, incomplete or unknown. The financial state describes whether funds are held, settled, released or awaiting reconciliation. A failed answer with trustworthy usage can still have a cost. A successful answer doesn't settle every earlier attempt.
Each supplier attempt needs its own record under the original request ID. Useful fields include the provider, submission stage, upstream request ID when available, terminal result, trusted usage and known or unresolved cost. I avoid storing prompt or response bodies just to diagnose retries.
The practical benefit is being able to explain a request without silently treating an unknown charge as zero.
Limit where a second attempt is allowed
In Motiontator's current routing contract, an alternate attempt is available only for stateless, non-streaming text without tools, when the routing policy enables it. It has budget and timeout limits and uses at most two different providers.
Streaming, tool, stateful and image requests aren't automatically replayed or resumed at another provider. A health change can influence a later new request without making the interrupted request safe to replay.
These are implementation boundaries, not an uptime guarantee. A request can still fail when there is no eligible route.
Make the client behavior visible too
A server-side attempt limit can be undermined if a client automatically retries outside it. The client and gateway need an explicit agreement about duplicate submissions.
In this gateway, a repeated Idempotency-Key returns a conflict with the original request ID, including after completion. It doesn't replay the previous response body. The caller should inspect the original request rather than assume a conflict means it is safe to create a new submission with a different key.
That's a product-specific contract; an OpenAI-shaped request and response don't establish identical retry semantics across providers.
What I would test before relying on failover
- A timeout before submission and a timeout after submission have distinct outcomes.
- A successful fallback doesn't erase unresolved cost from an earlier attempt.
- An interrupted stream isn't quietly replaced with a new generation.
- Tool requests don't enter the automatic replay path.
- A duplicate client submission points back to the original request.
- Cancellation doesn't display an unsupported promise of zero upstream cost.
The useful result is a request history that a developer can reconcile with what their application actually received.
I'm the developer of Motiontator. Its public routing and compatibility notes describe these limits. This article explains the design; it doesn't claim a measured reduction in outages or customer costs.
If you run an LLM application, how do you represent a request whose upstream execution is still unknown after a timeout?
Top comments (1)
The distinction between application outcome and upstream billing state is especially important. A timeout is evidence that the gateway lost visibility into the request, not evidence that the provider stopped processing it.
I like the approach of keeping individual provider attempts under one logical request ID, particularly when fallback succeeds but an earlier attempt remains financially unresolved.
One extension I'd find useful is separating the logical request record from the individual attempt records, with each attempt carrying its submission status, observed execution state, usage evidence, cost reconciliation status, and the reason another attempt was or wasn't permitted.
That could make it possible to report both the cost of the answer delivered to the user and the total known or potentially billable cost of fulfilling the request, without treating unknown amounts as zero.
I'd also be interested in how you'd handle late-arriving provider usage records. Would you preserve the original request receipt and append reconciliation events, or issue a revised receipt with an explicit version history? That seems important for auditing costs without silently rewriting the original evidence.