A request times out. The obvious next step is to try again.
For a video generation request, that can be an expensive guess. The provider may already be rendering the first video. You just didn't receive its task ID.
I'm working on AI Effect Studio, where people choose an effect, upload photos, and get a short video. The generation backend has to deal with this gap between sending a paid request and knowing what happened to it.
Here's how the current implementation handles it, including the part that still needs manual review.
The timeout doesn't tell you enough
Imagine this sequence:
- Our server sends a generation request.
- The provider accepts it and starts work.
- The connection drops before our server receives the response.
From our side, the request failed. From the provider's side, there's a video being generated.
Sending the same payload again doesn't necessarily resume that work. Unless the provider supports a documented idempotency mechanism, it may create another paid task.
Exponential backoff changes when that second request happens. It doesn't make the second request safe.
Record the attempt before sending it
Each local job stores a submission timestamp separately from its provider task ID. Before calling the generation API, the worker records that it's about to submit.
That gives the next worker enough information to distinguish these cases:
| What the job has | What the worker does |
|---|---|
| No submission marker and no provider ID | Claim the job, record the attempt, then submit |
| A provider ID | Query that existing task |
| A submission marker but no provider ID | Hold the job for support review |
| An output URL already recorded | Continue processing or saving that video |
The database updates are conditional, and processing uses a short lease so overlapping workers don't casually pick up the same job. The submission marker is checked and written in the database before the external call.
There's an awkward consequence here. If the worker crashes after writing the marker but before sending the request, the job also ends up waiting for review. It might never have reached the provider.
I accept that tradeoff for the expensive generation step. The marker prevents an automatic second submission, but it cannot prove that the first one happened. This is a conservative fallback, and it leaves some real work for support.
A provider-side idempotency key would be a better way to close that gap. A local timestamp can't give us exactly-once execution across two systems.
Retry the stage that failed
A video job keeps going after the model finishes. Depending on the effect, there's a soundtrack step, followed by saving the output to our own storage.
If storage fails, generating another video would be a costly way to retry a file transfer. The job records the source video URL so recovery can continue from the output it already has.
The soundtrack step has a different retry policy. It's much cheaper than generation, so it gets bounded retries. If those are exhausted, the delivery path can fall back to the silent footage. That is a product compromise: the user gets a video, but the soundtrack may be missing. The interface and product description need to make that possibility clear.
Credits need the same care. A confirmed generation failure restores the user's allowance. An ambiguous submission stays reserved while it's investigated, because a timeout alone doesn't tell us whether paid work is still running.
The question I use when reviewing these paths is pretty mundane: if this line runs twice, do we check the same work, or buy more work?
Polling an existing task, copying an existing output, and starting a fresh render deserve different answers. Putting all three behind one generic retry helper would hide the distinction I most need to see.
Top comments (1)
"If this line runs twice, do we check the same work, or buy more work?" is a good review question. I'd add one: does our own counter see the first run?
We have the cheap version of your problem in a mobile app, with chat and text-to-speech calls going through our own proxy, and we made two different choices. A chat call that times out is retried with backoff: a duplicate costs a fraction of a cent, and the app only uses one answer (a late answer from an earlier session is dropped). The speech call gets exactly one attempt with a long timeout, because a retry would arrive after the moment it was needed.
What I only noticed later: our client-side usage log records a call when the response comes back. A call that timed out after the provider finished it is paid but never counted. For us that's small. For a video render it's your whole "submission marker but no provider ID" row, which is why I like that you write the marker before the call, not after.