Retries are commonly treated as a simple reliability feature:
request failed -> try another provider
For AI and coding-agent traffic, that rule is incomplete.
The important question is not only whether the upstream returned an error. It is whether anything already reached the downstream client.
Before output
A retry may be safe when:
- no text was streamed
- no tool call was emitted
- no final usage was recorded
- the downstream connection is still active
After partial output
A retry may duplicate:
- text
- tool-call arguments
- tool execution
- usage records
- billing
This is especially risky when the first attempt failed after partial streaming.
State worth recording
A gateway should know:
- selected route
- attempt number
- whether downstream output started
- tool-call progress
- disconnect or cancellation state
- whether final usage arrived
- which usage source was used for settlement
Retry policy should be based on request state, not only an HTTP status code.
Reliability without state tracking can become duplicate output and duplicate billing.
Top comments (0)