Short answer: make the idempotency key a durable record owned by your database, and treat video generation as a state transition rather than a fire-and-forget retry. For a fintech upload pipeline, that keeps one thumbnail job attached to one audit trail while moderation coverage remains measurable.
The decision note
| Design | Duplicate protection | Moderation coverage | Operational cost |
|---|---|---|---|
| Client-generated key plus unique database row | Strong across process restarts | Explicit and auditable | Low |
| Server timestamp or request hash | Fragile under concurrent retries | Easy to misattribute | Medium |
| Queue-only deduplication | Depends on broker retention | Hidden after redelivery | Medium to high |
I use the first design for a one-person SaaS. It lets me ship weekly because the invariant is boring and visible: one (account_id, idempotency_key) row, one source object, one generation result. The revenue-per-hour test matters here. A clever queue plugin that nobody can inspect at 2 a.m. is a liability.
The catch is scope. A key cannot make an external provider idempotent if that provider accepts two independent jobs. Your ledger still prevents your API from creating the second job, but you must reconcile an uncertain handoff (more on that below). Keep the key for at least as long as a client can retry, and longer than your longest moderation review window.
What should a retry-safe video request record?
The record needs enough information to answer a support question without replaying a paid operation: who asked, which bytes were addressed, which policy version ran, and what response was returned. Store a content digest, requested thumbnail dimensions, moderation policy identifier, and a response snapshot. Do not store raw card data in this table; retain a pointer to the governed media store instead.
I model the lifecycle as accepted -> processing -> moderated -> ready or rejected. A retry of accepted returns the original receipt. A retry of ready returns the same thumbnail URL and moderation decision. A request that reuses the key with a different payload is a 409 conflict, because silently changing the video would turn an audit record into fiction.
Here is the core transaction in TypeScript. The provider call happens after the row is committed, so two HTTP workers cannot both claim a new request.
type RequestState = "accepted" | "processing" | "moderated" | "ready" | "rejected";
async function acceptVideo(input: {
accountId: string;
key: string;
sourceDigest: string;
width: number;
height: number;
policyVersion: string;
}) {
return db.transaction(async (tx) => {
const existing = await tx.mediaRequest.findUnique({
where: { accountId_key: { accountId: input.accountId, key: input.key } }
});
if (existing) {
if (existing.sourceDigest !== input.sourceDigest || existing.width !== input.width || existing.height !== input.height) {
throw new Error("IDEMPOTENCY_PAYLOAD_MISMATCH");
}
return existing.receipt;
}
const row = await tx.mediaRequest.create({
data: { ...input, state: "accepted" as RequestState, receipt: crypto.randomUUID() }
});
await tx.outbox.create({ data: { requestId: row.id, event: "GENERATE_THUMBNAIL" } });
return row.receipt;
});
}
The outbox worker claims an event with a lease, records the provider job identifier, and renews the lease while polling. If the worker dies after submission but before recording the identifier, the reconciliation query should search by your own correlation token before submitting again. If the upstream API offers no correlation lookup, mark the row processing and surface it for an operator decision; guessing creates the duplicate you were trying to prevent. That operator view should show the source digest, account, policy version, last heartbeat, and a link to the raw provider response. It also needs a deliberate action such as confirm-complete or abandon-with-reason, each recorded as an audit event. This sounds like extra product work until a compliance review asks why two near-identical thumbnails were delivered, at which point a timestamp and a vague log line are not evidence. I would rather spend one afternoon on that screen than lose a full day tracing a retry storm through three services.
How do retries, moderation coverage, and video thumbnails interact?
Fintech media is a policy pipeline, not a single render call. A video may produce several candidate frames, while moderation checks only a subset. Define coverage before optimizing latency: for example, sample the first frame, scene-change frames, and the final frame, then record the sample rule beside the result. The exact policy belongs to your risk team; the engineering requirement is that a repeated request reuses the same policy version and sample set.
Timeouts are ambiguous. The client may see a network timeout after the server committed accepted. Return the stored receipt on the next attempt, and expose a status endpoint keyed by that receipt. Avoid treating a timeout as permission to start over.
I once assumed a five-second client timeout was harmless. Then a mobile connection dropped just after commit, and the retry arrived while the first worker was still pulling the source object. The database constraint did its job; the missing status screen did not. A small GET /media/requests/{receipt} endpoint and a clear processing state removed the support loop.
Ship it.
Three metrics are enough to start: duplicate-key conflicts, age of the oldest processing row, and the percentage of generated thumbnails covered by the active moderation rule. Log the key as a pseudonymous hash, never as a raw account secret. Your mileage may vary on retention, especially when deletion requests must remove source media while preserving a financial audit event.
Where the runner-up design is better
Queue-only deduplication can be the right choice when jobs are disposable, the broker guarantees a bounded delivery window, and a duplicate render has no regulatory consequence. A content-addressed object store is also attractive for public, immutable videos: identical bytes naturally map to one artifact.
Stick with a database ledger when the request crosses an account boundary, triggers moderation, or appears on an invoice. The ledger adds a write and a cleanup policy. That is real complexity, and it is not suitable for a throwaway batch script. For an API that promises clients a stable receipt, though, the extra row is cheaper than explaining two thumbnails for one upload.
Top comments (0)