Two finalizers, one payment: what breaks when you bill for a GPU render

The whiteboard version has a bug in every box. I've been running TalkPix — a photo-to-talking-video tool — in production for a few months, and the interesting failures were never in the model. They were in the plumbing around it. Here are the three that cost real money, and what the fixes actually look like.

1. You will finalize the same job twice Long renders don't fit in a request. You start the prediction, hand the provider a webhook URL, and return. The webhook fires when the render finishes and your handler does the real work: parse the output, copy the file somewhere permanent, charge the account.

Then the webhook doesn't fire. Provider hiccup, a deploy mid-flight, a 500 from your own handler that never gets retried. So you add a poller as a fallback — a job that sweeps non-terminal rows and asks the provider directly.

Now you have two finalizers for one job, and they will eventually run at the same time. The webhook arrives while the poller is halfway through the same row. Both parse the same output. Both copy the same file. Both charge the customer.

The instinct is to guard with a read:

const job = await db.from("jobs").select("status").eq("id", id).single();
if (job.status === "completed") return;      // ← the race lives here
await chargeCredits(job);
await db.from("jobs").update({ status: "completed" }).eq("id", id);

That check-then-act is two round trips with a gap in the middle, and the gap is exactly wide enough for the other finalizer. It makes double-charging rarer, which is worse than not fixing it — the bug survives to production and shows up as a support ticket you cannot reproduce.

The fix is to stop asking and start claiming. Make the state transition itself the lock: a conditional UPDATE that only succeeds from a non-terminal state, and let the database tell you whether you won.

const claim = await db
  .from("jobs")
  .update({
    status: "completed",
    output_url: uploaded.publicUrl,
    credit_cost: finalCredits,
    completed_at: new Date().toISOString(),
  })
  .eq("id", job.id)
  .eq("prediction_id", prediction.id)   // this attempt, not a stale retry
  .in("status", ["queued", "processing"])  // ← the compare-and-swap
  .select("id");

if (!claim.data?.length) {
  // Someone else finalized this job. Not an error. Just stop.
  return "completed";
}

await reconcileCredits(job, finalCredits);  // now provably once

Two things earn their keep here. .in("status", [...]) is the compare-and-swap: only the transition out of a non-terminal state succeeds, so exactly one caller gets a row back. And .eq("prediction_id", …) scopes the claim to this attempt, so a webhook for a retried-and-abandoned prediction can't complete a job that has since moved on.

Everything before the claim — parsing, uploading — is idempotent by construction. Doing it twice wastes a few seconds of bandwidth. Everything after the claim runs exactly once, and that's where the money is.

The general shape: do the expensive-but-safe work optimistically, put the CAS immediately before the irreversible part.

1. You don't know the price until after you've paid it Billing per second of output is easy when you know the duration up front. With generative video you often don't. The user types a script; the model decides how long the speech takes; the file lands somewhere between four and twenty-something seconds. You cannot charge on the way in, and you cannot render for free on the way out.

So you do both, in two phases:

Reserve on submission: estimate high, hold the credits, reject if the balance can't cover it. The estimate has to be an upper bound, because a reservation that turns out to be too small means you rendered something the customer couldn't pay for.
Reconcile on completion: measure the real duration of the delivered file, compute the true cost, and release the difference back.
Both halves are Postgres functions, not application code:

select reserve_credits(p_user := $1, p_amount := $2, p_job := $3);
select reconcile_generation_debit(p_job := $1, p_actual_seconds := $2);
select refund_credits(p_job := $1);

Putting them in the database isn't ceremony. A balance mutated from application code is a read-modify-write across the network, which is the same race as §1 with worse consequences — and once you have two writers (a webhook and a poller, say) you need the arithmetic to be atomic with the balance check. Make the RPCs idempotent too, keyed on the job: a refund that has already happened should be a no-op that returns success, not a second refund.

One detail worth stealing: measure the duration from the file you actually delivered, not from what the provider claims. The two disagree more often than you'd like, and the file is the thing the customer received.

1. Provider errors are not your error messages This one embarrassed me. When inference fails, the provider hands back a message. The fast thing to do is put it on the screen — it's already a string, it's already about this job.

Then a billing error surfaced, and a dozen paying customers saw a message containing the provider's billing URL. From their side, our product was telling them to go top up an account with a company they'd never heard of.

Provider messages are diagnostics, not copy. They leak your vendor, they leak internal states, and they're written for you, not for the person who just uploaded a photo of their dog. The fix is a boundary: classify the raw error, keep the original in the database for your own dashboards, and render a message you wrote.

export function sanitizeProviderMessage(raw: string | null): string | null {
  if (!raw) return raw ?? null;
  if (!PROVIDER_INTERNAL_RE.test(raw)) return raw;    // safe, pass through
  const status = Number(raw.match(/\b(\d{3})\b/)?.[1] ?? 0);
  return userFacingProviderMessage(classifyProviderHttpError(status, raw));
}

The raw text still goes to the admin view. The customer gets one of a handful of sentences that say what happened and what to do about it.

While you're in there: classify errors before you retry, and never retry a thrown fetch. A request that failed after the provider accepted it may have started a render you're about to pay for twice.

Bonus: the failure that isn't one
Renders take two to four minutes. A CDN in front of your app will time out long before that — Cloudflare gives up around 100 seconds and returns a 524. If your client treats "the request died" as "the job died", it will report a failure while the render finishes normally and the webhook completes it thirty seconds later.

Recover on the gateway timeouts — 520 through 530 — by going back to polling job state. Do not recover on a 402 or a 500 from your own API; those mean what they say. And whatever you do, don't make the recovery path re-submit the job. Two renders, one payment, in the other direction.

The through-line
Every one of these is the same mistake wearing a different hat: treating a distributed operation as if it happened once, in order, on one machine. A webhook is not a function call. A balance is not a variable. A provider's error is not your UI.

The model is the easy part. It's a well-documented HTTP call that either returns a file or doesn't. The hard part is the ledger you wrap around it, and ledgers have been hard for a lot longer than diffusion models have existed.

I build TalkPix, which turns a photo and a script into a lip-synced talking video. It's pay-once — credit packs, no subscription — which is partly a product decision and partly because per-render billing forced me to get the accounting right early.