DEV Community: Indie Rob

I Made Claude Code Prove Billing End-to-End Before I Let It Ship

Indie Rob — Tue, 08 Sep 2026 07:17:14 +0000

I’ve been using coding agents for a while, but during the last release of a SaaS project I changed one thing about how I used them.

I stopped defining “done” as:

implementation finished, tests passing

and gave the agent an external condition instead:

Do not let me ship until you can prove that billing works end-to-end against real infrastructure.

That turned out to be much more useful than another round of code generation or code review.

Because the tests were already green.

The real system was not.

The release gate

The project uses FastAPI, Next.js, Postgres, Redis, Celery and Paddle.

Billing already had tests around checkout, webhook processing, subscriptions, credit grants and idempotency.

At that point it would have been very easy to say:

Looks good. Ship it.

Instead, I asked Claude Code to prove the actual money path.

That meant it couldn't stop after reading the code or running pytest.

It had to create an environment where a real Paddle sandbox transaction could go through the application.

So it provisioned temporary infrastructure on Render through the API:

Postgres
Redis
backend
frontend

Then it configured a temporary Paddle sandbox webhook destination pointing at that deployment.

The requirement was simple: act like a real customer and show me the resulting state.

A real checkout, not a mocked one

The agent then went through the actual application flow.

It authenticated through the app, opened checkout, completed a Paddle sandbox payment and inspected what happened afterward.

Not just the HTTP response.

Not just the redirect.

The resulting state.

It checked:

the Paddle webhook deliveries;
the local subscription record;
the credit ledger;
the resulting account state;
whether replaying a real webhook changed anything twice;
whether a second checkout was incorrectly allowed.

Before payment the account was free, with no paid subscription.

After payment it had the expected active subscription and the expected credit balance.

So far, good.

Then the sandbox exposed a bug.

The assumption hiding behind the code

We had implicitly been thinking about these events as a sequence:

subscription.created
        ↓
subscription.activated

That mental model was wrong.

In the sandbox, the two webhook handlers could overlap.

Both could begin while there was still no local subscription row.

Conceptually, each handler was doing something like:

subscription = find_subscription(provider_id)

if subscription is None:
    create_subscription(provider_id)

Perfectly reasonable if the operations happen sequentially.

Not safe if two different webhook events execute at the same time.

The actual failure mode was:

Handler A                       Handler B

SELECT subscription             SELECT subscription
→ nothing                       → nothing

INSERT subscription
                                INSERT subscription

success                         unique constraint violation

The database was correctly preventing duplicate subscriptions.

But one of the webhook handlers was now failing.

And this was not an idempotency problem in the usual sense.

These were two different Paddle events.

Making each individual event idempotent did not prevent two distinct events from racing over the same underlying subscription.

The deeper bug was our sequencing assumption.

Why the tests didn't catch it

Our tests had covered the individual operations.

They had covered webhook replay.

They had covered duplicate credit protection.

What they had not challenged was the assumption that two different subscription events would effectively be handled one after another.

That is the part I found interesting.

The code was tested.

The assumption connecting the pieces wasn't.

A real external system invalidated it almost immediately.

The fix

The fix was not to invent a more complicated application-level lock.

We already had something very good at arbitrating concurrent writes:

the database.

The final pattern was roughly:

try to insert the subscription;
let the unique constraint decide which concurrent transaction wins;
isolate the conflicting insert inside a SAVEPOINT;
roll back only that inner operation if the insert loses;
re-read the subscription row created by the winner;
continue processing normally.

The SAVEPOINT mattered.

Without it, handling the uniqueness collision could poison the larger transaction and interfere with the rest of the webhook processing.

The losing handler should not become an error path.

It should simply discover:

another handler created the same logical subscription first

and continue from the state that now exists.

In simplified form:

subscription = find_subscription(provider_id)

if subscription is None:
    try:
        with session.begin_nested():
            subscription = create_subscription(provider_id)
            session.flush()
    except IntegrityError:
        subscription = find_subscription(provider_id)

The production code has more state handling around this, but that is the important idea.

The unique constraint becomes part of the concurrency mechanism rather than merely an exception waiting to happen.

Then I made the agent prove the fix

This is the part I now care about most.

After changing the code and adding the regression test, the task was still not finished.

Claude redeployed the staging environment and ran another actual sandbox payment.

This time the webhook events arrived in the opposite order.

That was useful.

The fix was no longer depending on the ordering we happened to see during the first run.

The resulting subscription state was correct and credits were granted exactly once.

I then replayed the real webhook notification and checked that the state and credit ledger remained unchanged.

Only after that did I consider the billing path proven.

The temporary Render infrastructure was deleted afterward, along with the temporary Paddle webhook destination. Cleaning up the receiver without cleaning up the sender would have left Paddle retrying a dead endpoint.

“The API returned 200” is not the success condition

A comment on a Reddit discussion about this gave another good example.

Someone described an agent-run publishing pipeline that considered a post successful because the publishing API returned 200.

But sometimes the published content had been silently truncated.

The fix was to make the agent fetch the published item back and diff it against the source.

That is the same pattern.

These are different claims:

the action completed without error

and:

the intended state now exists in the real system

The second one is what we usually care about.

Yet a lot of automation, including agentic automation, stops at the first.

Coding agents need falsifiable completion conditions

This changed the way I think about using coding agents.

A prompt like:

Implement Paddle billing.

is useful.

A prompt like:

Implement Paddle billing and write tests.

is better.

But the much more interesting version is:

Here is the external condition that would prove billing actually works. Keep going until you can demonstrate it.

That produces a different workflow:

implement
    ↓
deploy
    ↓
interact with the real system
    ↓
inspect resulting state
    ↓
try to falsify the assumptions
    ↓
fix
    ↓
repeat the proof

The agent is no longer just responsible for producing code.

It is responsible for producing evidence.

I started applying this elsewhere

Once I started thinking this way, I found the same problem in several other parts of the release.

Run the setup guide from a truly fresh clone

A setup procedure can work perfectly on the developer's machine because the shell already contains environment variables, dependencies or state that the documentation forgot to mention.

So don't ask:

Are the getting-started docs correct?

Make the agent actually start from a clean clone and execute them.

Clear ambient environment variables

One of our setup paths appeared reproducible until we removed configuration inherited from the agent's shell.

The environment had been silently helping the test pass.

The interesting test wasn't:

does setup work here?

It was:

does setup work with none of our accidental local state?

Ask the database whether migrations match the models

The application tests were green.

A fresh:

alembic check

reported 174 schema operations.

Some were metadata details rather than catastrophic defects, but the important point was that “tests pass” and “the deployed schema matches the declared model” were separate claims.

So we tested the second one directly.

Prove regression tests by restoring the bug

An agent can write a regression test that passes without really protecting against the original failure.

A simple way to challenge that is:

apply the fix;
confirm the new test passes;
temporarily restore the old broken behavior;
confirm that the new test fails;
restore the fix.

That gives you evidence that the test is detecting the actual regression rather than merely exercising nearby code.

Agents are good at closing loops

There is a lot of attention around how many lines of code an AI agent can write.

I'm finding the more valuable property is something else.

They can stay inside verification loops that are tedious enough that humans often stop early.

Provision the environment.

Deploy.

Exercise the system.

Inspect the database.

Read webhook deliveries.

Replay the event.

Change the implementation.

Deploy again.

Repeat the transaction.

Compare the result.

Clean everything up.

None of those individual steps is especially impressive.

Doing the entire loop without deciding that “probably fine” is good enough is where the value starts to appear.

This came out of building The Fabrica

All of this happened while preparing the first release of The Fabrica, a production FastAPI + Next.js SaaS foundation I built after getting tired of rebuilding the same infrastructure for every product.

It includes the parts that tend to consume the beginning of a SaaS project:

authentication;
Paddle subscriptions;
credit accounting;
background workers;
database migrations;
lifecycle and deletion handling;
production-oriented setup and deployment patterns.

The point isn't that those components are difficult to generate.

Coding agents can generate most of them very quickly.

The difficult part is knowing whether all of those components still behave correctly once they interact with each other and with real external systems.

That is what I wanted to package.

The first release is here:

https://www.thefabrica.dev/?utm_source=devto&utm_medium=article&utm_campaign=fabrica_launch_010&utm_content=dev_billing_e2e

And the main thing I took from the release process is this:

Don't only ask your coding agent to build the thing. Give it a condition that reality can prove false.

credits_remaining -= 1 is a race condition

Indie Rob — Thu, 03 Sep 2026 21:49:01 +0000

A credits_remaining: int column on your users table looks perfectly fine for an AI SaaS. Sign a user up, drop 100 credits into their row, deduct on each request, top up on payment. Done.

It stays fine until the work you're charging for goes asynchronous. Then it becomes surprisingly easy to charge twice, overspend, or lose track of why a balance changed.

I built it the naive way first. Here is the failure mode I ran into, and the pattern I ended up shipping.

The race

Imagine a user has 10 credits. Two API requests arrive almost simultaneously, each costing 7 credits' worth of compute.

Both handlers read the balance:

balance = await db.fetch_val(
    "SELECT credits_remaining FROM users WHERE id = :id",
    {"id": user_id},
)
# 10

Both check 10 >= 7. Both decide the user can afford the job. Both enqueue the work.

You just sold 14 credits of compute for 10.

If your compute costs real money — a hosted LLM call, a GPU minute, a paid third-party API — you are eating the delta. If it costs enough, that is a real leak.

"But I'll wrap it in a transaction"

Common first fix. Doesn't do what most people think.

Postgres defaults to READ COMMITTED isolation. Under READ COMMITTED, two concurrent transactions can each see the pre-spend balance before either commits. Both read 10, both decide 7 is affordable, both write back a new balance of 3 — the second overwriting the first. Net effect: same overspend, and you have thrown away one of the deductions on top of it.

SERIALIZABLE isolation would catch this. It also turns every write in your application into something that might fail with a serialization error and need a retry loop, whether it has anything to do with the billing race or not. Most teams don't want that as a global setting.

The narrower fix: lock the row for the spend

The tighter answer is to serialize only the read-and-write for the balance being spent from.

async with db.transaction():
    balance = await db.fetch_val(
        "SELECT credits_remaining FROM users WHERE id = :id FOR UPDATE",
        {"id": user_id},
    )
    if balance < cost:
        raise InsufficientCredits()
    await db.execute(
        "UPDATE users SET credits_remaining = credits_remaining - :cost "
        "WHERE id = :id",
        {"id": user_id, "cost": cost},
    )

SELECT ... FOR UPDATE acquires a row-level lock. The second concurrent transaction blocks until the first commits, then sees the updated balance of 3 and correctly refuses the second 7-credit job.

That closes the race. Two different jobs can no longer overspend the same balance.

Another problem shows up almost immediately.

The second race: retries

Workers retry. Brokers redeliver. HTTP clients retry on 504. Users double-click. If the same logical job can enter your deduction path twice, your billing correctness now depends on every caller in the system behaving perfectly and never retrying a completed request.

That is a bad invariant to bet a billing system on.

The row lock doesn't help here. Each retry is its own transaction. It dutifully takes the lock, sees the current balance, and deducts again.

The fix is separate: an idempotency key at the deduction layer. Give every logical job a stable ID, and make the deduction insert unique on (user_id, job_id, kind). A retry that tries to re-insert the same fact gets rejected by the unique constraint, and the handler treats that rejection as "already recorded, carry on".

The distinction I originally missed

Row lock and idempotency key are both about "the same fact shouldn't be charged twice", but they solve different races:

Row lock prevents two different jobs from overspending the same balance.
Idempotency key prevents the same job from being charged twice.

I originally treated them as one problem. They aren't. You need both, and they compose cleanly in the same handler.

The pattern I ended up shipping: an append-only ledger

Once I accepted that "the balance" is derived state, not primary state, the schema simplified.

CREATE TABLE credit_entries (
    id          BIGSERIAL PRIMARY KEY,
    user_id     UUID NOT NULL REFERENCES users(id) ON DELETE CASCADE,
    delta       INTEGER NOT NULL,
    kind        TEXT NOT NULL,
    job_id      UUID,
    reason      TEXT,
    created_at  TIMESTAMPTZ NOT NULL DEFAULT NOW(),
    UNIQUE (user_id, job_id, kind)
);

CREATE INDEX ON credit_entries (user_id, created_at);

Every movement is a row:

grant           +100
deduct            -7
task_started       0
task_completed     0
refund            +7

The balance is SUM(delta) WHERE user_id = ?. Never stored, always computed. If you're worried about the read cost, cache it — but the ledger stays the source of truth.

Spending looks like this:

async with db.transaction():
    balance = await db.fetch_val(
        """
        SELECT COALESCE(SUM(delta), 0)
        FROM credit_entries
        WHERE user_id = :id
        FOR UPDATE
        """,
        {"id": user_id},
    )
    if balance < cost:
        raise InsufficientCredits()
    try:
        await db.execute(
            """
            INSERT INTO credit_entries
                (user_id, delta, kind, job_id, reason)
            VALUES
                (:id, :delta, 'deduct', :job_id, :reason)
            """,
            {
                "id": user_id,
                "delta": -cost,
                "job_id": job_id,
                "reason": reason,
            },
        )
    except UniqueViolation:
        # (user, job, 'deduct') already exists. Retry is a no-op.
        pass

Row lock handles the concurrent-jobs race. Unique constraint handles the retry race. Same handler. Both problems addressed, no special cases.

Why the ledger earns its weight

More machinery than a single integer column, obviously. It earns it in ways you feel every week once you have real users.

Refunds are new facts, not mutations. A failed job gets a +7 refund entry that references the original deduction. The original deduction stays. Nothing was rewritten. You can look back at the sequence and see exactly what happened.

Failed jobs remain explainable. The task_started and task_completed entries mean you can tell the difference between "we charged the user and the job succeeded" and "we charged the user and something died".

Reconciliation becomes possible. If workers crash between deducting and completing, you can find orphaned deductions programmatically — deduct entries without a matching task_completed or refund after some timeout — and repair them without opening the database by hand.

Support can answer "why is my balance 37?" They run one query and read the last N rows in chronological order. This is a real support cost you're paying whether you notice it or not.

Billing state has an audit trail by default, without you having to bolt on a separate audit log later.

When it is overkill

If your credits map 1:1 to synchronous API calls, no async work, no retries, no long-running compute, then no — you don't need this. A locked update on a column is fine.

The ledger earns its complexity when at least one of these is true: work happens off the request path, workers can retry, refunds are non-trivial, or support has to answer questions about historical balances.

If you're building an AI SaaS with billed LLM calls, all four are usually true from month one.

In production

I put the decision doc, a concurrency test receipt (real PostgreSQL sessions, two concurrent deducts of 7 on a starting balance of 10, asserting exactly one succeeds and the balance ends at 3), and a few other engineering excerpts into a public inspection repo for The Fabrica:

github.com/webrot9/thefabrica-inspect

Not the commercial source. Enough code and enough decisions to inspect whether the engineering claims hold up before deciding to license the full codebase. The ledger is one of the pieces I most wanted to expose there, because it's exactly the kind of thing that separates a starter kit from a codebase you can actually run a business on.

If you're building anything with async, billed work, do the ledger from day one. Retrofitting it after your first "sold 14 for 10" incident is not fun.