A safety check can fail before a media provider has received a generation request. Your queue needs to represent that state separately from an uncertain generation submission. Otherwise, a visitor can lose a concurrency slot indefinitely even though there is no generation task to poll.
We encountered this while operating Rendastic, a Go and SQLite application that submits video and image jobs to external providers. Several jobs stayed in a pending-confirmation state after an automated content check stopped returning a usable result. Restarting the application preserved the stuck state. Retrying the same paid POST could introduce a second charge.
Here is the state-model change we used, and how to test it without generating media.
Keep customer reservations and provider uncertainty separate
A request can reserve three different things:
- A customer's free attempt or credits.
- A local concurrency slot.
- A conservative budget for an external provider request.
These reservations have different lifecycles. When a required safety check cannot finish, we can fail the customer job and release the first two. The external request may still have incurred a charge, so its journal entry and uncertain cost reservation remain unchanged.
Releasing a customer's allowance does not prove that the provider refunded anything. That distinction matters when reconciling costs later.
Use distinct error states
Our original generic uncertainty path was useful for a media submission that might have reached the provider. It was insufficient for a failed safety check with no pollable task identifier.
We introduced a typed moderation-unavailable error and gave content checks a bounded deadline. The following is illustrative pseudocode for the decision:
if required_check_is_unavailable:
withhold_unchecked_result()
fail_customer_job_and_release_customer_reservation()
preserve_original_provider_journal()
elif main_generation_submission_is_uncertain:
hold_customer_job_for_reconciliation()
preserve_original_request_key()
The first branch concerns technical failures: timeout, transport failure, or an unusable safety verdict. An explicit policy refusal is a separate business outcome and should have its own disclosed handling.
No unchecked result becomes downloadable in either branch. An uncertain external request keeps its original identity; it is never retried under a fresh key merely to escape the uncertain state.
Commit the terminal transition and release together
The customer's job status and quota or credit release need to change in one database transaction. A process crash between those operations can otherwise leave a failed job that still consumes quota, or a running job whose credits were already returned.
The operation also needs to be idempotent. Two workers can observe the same failure, and maintenance can run after a restart. A terminal-state guard should ensure that repeating settlement neither returns credits twice nor changes an already-settled result.
SQLite transactions are useful here because job admission, local quota, and the credit ledger live in the same database. Systems using separate services need an equivalent reconciliation mechanism rather than assuming cross-service writes commit together.
Recover old jobs conservatively
We restricted historical recovery to identifiable, stale moderation requests with no task ID and no unresolved main media request. A job with evidence of an accepted or uncertain media submission stayed in reconciliation.
This scope avoided treating every old pending job as safe to release or replay. We compared the provider journal before and after recovery to confirm that repairing the customer state did not create new external requests or erase uncertain provider costs.
Test failure paths without paying for media
A local fake provider can return a timeout, a pending response, a valid refusal, or a completed response. The important assertions are behavioral:
- An unavailable safety check releases the customer's reservation and concurrency slot.
- No unchecked output is served.
- Reprocessing the same request does not issue another paid POST.
- Settling the same failure twice does not return credits twice.
- An uncertain main generation task remains held for reconciliation.
- Recovery after restart preserves the original external request journal.
Successful generation tests alone do not cover these boundaries. A useful acceptance run should explicitly interrupt each stage where customer state and provider state can diverge.
Disclosure: I operate Rendastic, where we implemented this recovery flow. This article describes our application behavior, not a guarantee about another provider's billing or moderation decisions. This draft was generated by an AI agent and checked against our implementation and recorded tests.
Top comments (0)