DEV Community: Cuong Duong

Two ways usage-based AI billing quietly loses money

Cuong Duong — Sun, 06 Sep 2026 07:12:08 +0000

If you charge for tokens, two bugs will take real money off you before any customer notices: charging before you know the token count, and charging twice when Stripe retries a webhook. The fixes are small — meter after the stream closes using the provider's own usage numbers, and make the ledger reject duplicates at the database level instead of in your code.

Both bugs are invisible in testing. Streams complete, webhooks arrive once, and the numbers look right. They show up in production, on the requests that get cancelled and the webhooks that get redelivered.

Bug one: metering before the answer exists

The tempting order is to estimate cost up front, deduct, then stream. It's tempting because the balance check is already there and an estimate feels close enough. It isn't: output length varies by an order of magnitude on the same prompt, and the moment you add a model with different pricing, every estimate silently becomes wrong.

Do it the other way around. Check the balance before the call — a cheap gate, not a charge — then stream, then read the usage the provider reports on the final event and meter that.

// gate, don't charge
    if (await balance(customerId) <= 0) return payWall();

    const stream = await client.messages.stream({ ... });
    const final  = await stream.finalMessage();

    const { input_tokens, output_tokens } = final.usage;
    const cost = ceilToCredit(
      input_tokens  * MODEL.inPrice +
      output_tokens * MODEL.outPrice
    );
    await meter.record(customerId, cost);   // after, with real numbers

Round up to your credit unit, and store credits as integers. Floating-point money is a slow leak that only shows up when you reconcile at month end, and a credit worth a tenth of a cent makes rounding up honest rather than greedy.

Two edge cases decide whether this holds: if the client disconnects mid-stream you still owe the provider for the tokens produced, so meter what was generated, not what was delivered; and if the metering call itself fails after a successful response, write the pending charge somewhere durable and retry it, rather than dropping it and calling that customer service.

Bug two: the webhook that arrives twice

Stripe retries a webhook whenever it doesn't get a fast 2xx — a timeout, a deploy, a cold start. If a retry runs your "add credits" handler again, the customer gets a second top-up for one payment. The version of this that hurts is the one nobody reports.

An if (alreadyProcessed) check in application code doesn't fix it, because two retries can run concurrently and both read "no" before either writes. Push the guarantee down to the database: make the payment's own identifier a unique key on the ledger.

-- one row per checkout session, enforced by the database
    CREATE UNIQUE INDEX credit_ledger_session_uniq
      ON credit_ledger (stripe_checkout_session_id);

    -- handler
    try {
      await db.insert(creditLedger).values({ ... });
    } catch (e) {
      if (isUniqueViolation(e)) return res.status(200).end(); // already credited
      throw e;
    }

The duplicate now fails at insert time, under a lock the database is already taking, and you answer 200 so Stripe stops retrying. Same shape works for the reverse direction: one row per metered request, so a retried deduction can't double-charge either.

While you're there, check the concurrency case on spending too. Fire five requests at once against a balance of one and the naive read-then-deduct path lets all five through. A row lock on the balance (SELECT ... FOR UPDATE), or a single in-flight request per user, closes it — worth doing before you have the kind of customer who finds it on purpose.

Where it pays, and where it doesn't

Worth doing

Any product where the marginal cost of a request is real money and users control how much they consume. The two fixes together are an afternoon; one month of a heavy user on estimated billing costs more.

Not worth it

Flat-rate subscriptions with generous limits, or an internal tool. If nobody is billed per token, a token counter is telemetry, not accounting — keep it in your logs and skip the ledger.

The trade-off. Metering after the fact means a user can overshoot a near-zero balance by one request — accepted deliberately: refusing to answer mid-stream is worse product than eating a fraction of a cent. And a unique-key ledger makes replays harmless but not free; you still need the failed-metering retry path, or you've moved the leak instead of closing it.

Both patterns ship in my Next.js SaaS Starter Kit ($149 one-time, my own product) alongside Auth.js and Stripe Billing Meters — but they're a hundred lines each and worth writing yourself if you'd rather.

Regression-testing an agent whose output is never the same twice

Cuong Duong — Sun, 06 Sep 2026 07:11:35 +0000

You change one line of a prompt and have no idea what you broke, because diff is useless on free text. The fix is to stop comparing text: freeze a set of real traces, assert on behaviour — which tools got called, which constraints held, what the agent refused to do — and run that suite on every prompt or model change.

Concretely, three files and a command. A cases.jsonl of frozen inputs, an assertions.ts that checks behaviour rather than strings, and a runner you invoke after each change. If you work in Claude Code or Cursor, this is a good candidate for a slash command, so the check costs you one line instead of a context switch.

How it works

The unit under test is not the final message. It's the trace: the ordered record of what the agent did to produce that message — tool calls and their arguments, retrieved documents, the number of turns, tokens spent, and the final output. A trace is structured data, and structured data can be asserted on.

Four kinds of assertion cover most real failures:

Sequence — the agent called search before answer, and never called delete_record at all. Cheap, deterministic, catches the scariest class of regression.
Constraint — the answer cites at least one retrieved document; the JSON parses against the schema; the reply stays under the token budget.
Refusal — for inputs that should be declined, the agent declines. Prompt edits aimed at making an agent more helpful loosen refusals as a side effect, every time.
Judged — a second model scores the output against written criteria ("does it answer the question actually asked?"). Use it last and sparingly; it's the only assertion that can itself be wrong.

{ "id": "refund-past-window",
      "input": "I want a refund, bought 400 days ago",
      "assert": {
        "tools_called":     ["lookup_order", "check_policy"],
        "tools_forbidden":  ["issue_refund"],
        "must_mention":     ["policy", "30 days"],
        "max_output_tokens": 300
      } }

Run the suite twice per change: once before, once after. What you report is not pass/fail but movement — three cases that used to call check_policy no longer do. That's the sentence you want in front of you before a deploy, and it's the one a screenshot of a happy-path chat will never give you.

Keep the case set small and adversarial. Twenty cases you chose because each one broke something once beats two hundred generated to look thorough — the suite has to run in under a minute or you will stop running it.

Where it pays, and where it doesn't

Worth doing

Prompts change weekly, models get swapped, the agent can take actions with consequences, or more than one person edits the prompt. Anywhere a silent behaviour change costs more than the hour this takes to set up.

Not worth it

A single-turn wrapper with no tools and no side effects — assert the JSON schema and move on. Also skip it while you're still changing the agent's job description weekly: you'd be freezing traces of a thing that doesn't exist yet.

The honest downsides. Behavioural assertions encode your current idea of correct, so a deliberate improvement shows up as a failure and you rewrite cases — budget for that. Judged assertions add cost and their own flakiness, and a judge model that gets upgraded under you is itself an untested change. And a frozen trace set drifts away from real traffic; refresh it from production logs every so often, or it slowly starts testing last quarter's product.

The alternative is not "no testing" — it's testing by user report, on a Tuesday, in public.

I package this workflow — eval suites, trace review, regression checks, red-teaming — as slash commands and sub-agents for Claude Code and Cursor. It's my own product, $29 one-time: AI Agent QA & Eval Toolkit. The approach above works fine hand-rolled, too.