DEV Community: Shipmind Labs

Extracting a Service From a Monolith Without a Code Freeze

Shipmind Labs — Mon, 17 Aug 2026 13:33:13 +0000

Every monolith extraction reaches the same question: how do you know the new service returns the same answers as the old code, on real production data, before anything depends on it? A test suite written against the new service only proves the new service is self-consistent. It proves nothing about the behaviour you inherited, which is undocumented, occasionally wrong, and already relied upon by callers who built around the wrongness.

We do this work regularly — Python 2 to 3 migrations, monolith-to-microservices rewrites of systems we did not write — and the constraint is always the same. There is no freeze. The monolith keeps shipping while the extraction runs, because the client's roadmap does not pause for our architecture diagram. That single constraint rules out the comfortable approach of branching, rewriting, and cutting over on a quiet Sunday. It forces a method where the old and new implementations run side by side in production for weeks, and the old one stays authoritative until the evidence says otherwise.

The spec is the old code's output

The first thing we stop doing is writing the new service from the ticket. The ticket describes what the endpoint is supposed to do. Production describes what it actually does: the rounding that happens one step earlier than the docs suggest, the field that is omitted rather than null when a relation is missing, the ordering that is technically unspecified but that a mobile client parses positionally anyway.

So we build the candidate service, deploy it, and give it no traffic. Instead, the monolith calls it out of band on real requests and compares the two answers. The candidate cannot affect the response, cannot add latency to it, and cannot fail the request. Those three properties are what make it safe to run this on a payment-adjacent read path in production rather than on a replayed sample that has already lost the interesting cases.

The dispatch is small on purpose.

import functools
import logging
import random
from concurrent.futures import ThreadPoolExecutor

log = logging.getLogger('shadow')
_pool = ThreadPoolExecutor(max_workers=8, thread_name_prefix='shadow')


def shadow_read(name, sample_rate, classifier):
    def decorator(legacy_fn):
        state = {'candidate': None}

        @functools.wraps(legacy_fn)
        def wrapper(*args, **kwargs):
            authoritative = legacy_fn(*args, **kwargs)
            candidate = state['candidate']
            if candidate is not None and random.random() < sample_rate:
                _pool.submit(
                    _compare, name, candidate, classifier,
                    authoritative, args, kwargs,
                )
            return authoritative

        def register(candidate_fn):
            state['candidate'] = candidate_fn
            return candidate_fn

        wrapper.candidate = register
        return wrapper
    return decorator


def _compare(name, candidate, classifier, expected, args, kwargs):
    try:
        actual = candidate(*args, **kwargs)
    except Exception:
        log.warning('shadow.error', extra={'endpoint': name})
        return
    verdict = classifier(expected, actual)
    if verdict != 'match':
        log.warning('shadow.diff', extra={'endpoint': name, 'verdict': verdict})

Two details in there are not stylistic. The pool is bounded and the submit is fire-and-forget, so a slow candidate degrades into dropped comparisons instead of queued work piling up in the request path. And the comparison swallows every exception from the candidate, because a traceback escaping into the monolith's response is exactly the incident this whole exercise exists to avoid.

Comparing is where the actual work is

A naive equality check reports a diff on every single request and teaches the team to ignore the log within a day. Real payloads differ in ways that do not matter and in ways that matter enormously, and the value of the method comes entirely from separating the two automatically.

We normalise first, then classify what survives normalisation.

from decimal import Decimal


def normalize(payload):
    if isinstance(payload, dict):
        return {
            key: normalize(value)
            for key, value in sorted(payload.items())
            if value is not None
        }
    if isinstance(payload, list):
        return [normalize(item) for item in payload]
    if isinstance(payload, Decimal):
        return payload.quantize(Decimal('0.01'))
    if isinstance(payload, float):
        return Decimal(repr(payload)).quantize(Decimal('0.01'))
    if hasattr(payload, 'isoformat'):
        return payload.replace(microsecond=0).isoformat()
    return payload


def _keyed(items):
    return {item.get('id'): item for item in items if isinstance(item, dict)}


def classify(expected, actual):
    if expected == actual:
        return 'match'
    left, right = normalize(expected), normalize(actual)
    if left == right:
        return 'cosmetic'
    if isinstance(left, list) and isinstance(right, list):
        if _keyed(left) == _keyed(right):
            return 'ordering'
    if set(_flatten(left)) - set(_flatten(right)):
        return 'missing_field'
    return 'behavioural'

The classes are not decoration; they are a work queue. Cosmetic diffs mean the candidate is right and the serialiser needs alignment. Ordering diffs mean the old query had an implicit sort that a client is probably depending on, so we make it explicit in both. Missing fields are usually a null-versus-omitted disagreement, which is a one-line fix and a genuine breaking change if shipped unnoticed. Only the behavioural bucket needs someone to read the old code and decide whether the difference is a bug we are inheriting on purpose or a bug we just wrote.

We inherit deliberate bugs more often than people expect. If a report has undercounted a category for years and the client's own spreadsheets are reconciled against it, matching the old behaviour and raising it as a separate ticket is the correct engineering decision. Fixing it silently during an extraction turns an infrastructure change into a data incident that nobody will connect back to us for a week.

Cutover is a routing change, not a deploy

When a route's behavioural diff count has been zero for long enough to cover the slow paths — month-end jobs, retried webhooks, the tenant with the unusual configuration — reads move. We do this at the edge so that reverting takes seconds and touches no application code.

upstream monolith  { server monolith:8000; keepalive 32; }
upstream invoices  { server invoices:8000;  keepalive 32; }

map $http_x_cutover_group $invoices_backend {
    default    monolith;
    canary     invoices;
    everyone   invoices;
}

location /api/v1/invoices {
    proxy_pass http://$invoices_backend;
    proxy_set_header X-Request-Id $request_id;
    proxy_next_upstream off;
}

The header is set by an upstream layer that knows the tenant, so the group can widen gradually and a rollback is one map edit and a reload. We deliberately turn off retry to the next upstream here: silently replaying a request against the other implementation during a cutover is how you get a duplicate side effect and lose the ability to explain what happened.

Writes come last, and they do not get a shadow. Writing twice to two implementations means two sources of truth and a reconciliation problem we would have to build and then throw away. Instead the extracted service takes ownership of its tables in one step, and the monolith's write path becomes a thin client calling the service. That is the moment of real risk in the project, and the reason it lands late: by then, every read path has already proven the new service's model of the data matches the old one on production traffic.

What it costs to run

Honest accounting matters, because this method is not free and it is not always justified. Shadow reads add load to the database — the candidate usually runs its own queries against the same instance, so a heavy report endpoint gets sampled at a few percent rather than fully. The comparison log is verbose and needs a retention policy, and the classifier itself needs unit tests, because a classifier that quietly returns match on everything looks exactly like success.

The largest cost is not technical. While shadowing runs, every change the monolith team ships to a shadowed path has to be mirrored in the candidate, or the diff count rises for reasons unrelated to the extraction. That double-maintenance tax is bearable for weeks and corrosive over quarters, so we scope extractions to what a route group can clear in that window rather than announcing a rewrite of the whole system. A diff dashboard that has been flat and non-zero for a month is a signal to either finish the route or delete the candidate.

What this buys is a migration where nobody has to answer the question of whether the new service is correct with an opinion. The answer is a count, taken from production, on the traffic that actually exists — and the cutover is the boring part, which is where we want the risk in someone's live system to sit.

Originally published on shipmindlabs.com — where we write about payment systems, infrastructure and marketplace backends.

Testing Money Paths: Four Tests Our Review Gate Requires

Shipmind Labs — Sun, 16 Aug 2026 15:51:34 +0000

A payment service can pass every unit test it has and still lose money. The failures that matter in production are not about one request being wrong: a provider webhook delivered twice, two workers picking up the same payout row, a ledger that drifts by a cent until somebody closes the books and notices. They are about two correct requests colliding, or one correct request arriving again.

We have built payment processing, virtual account ledgering, and reconciliation flows across fintech and crypto for years, and one pattern holds everywhere: money bugs are found by accountants, not by monitoring. That is what makes them expensive. A 500 appears on a dashboard within a minute. A double payout appears in a bank statement, after the money has left. Rolling back the deploy restores the code, not the transfer.

So the money path gets a different test standard than the rest of the service. Four tests, all of them against a real database. This is what they look like, and what our review gate rejects when they are missing.

No mocked database on a money path

Money correctness does not live in application code. It lives in constraints, transaction boundaries, and lock semantics. A unique index is what actually makes an idempotency key idempotent. FOR UPDATE SKIP LOCKED is what actually makes a queue worker safe. A mocked session tests our belief about Postgres, and our belief is exactly the thing that is wrong when this class of bug ships.

So the suite starts a real database on the same major version as production. Everything below assumes that fixture.

import os
import pytest
from sqlalchemy import create_engine, text


@pytest.fixture(scope='session')
def engine():
    return create_engine(os.environ['TEST_DATABASE_URL'], future=True)


@pytest.fixture
def db(engine):
    with engine.begin() as conn:
        conn.execute(text(
            'TRUNCATE ledger_entries, transfers, payout_jobs RESTART IDENTITY CASCADE'
        ))
    with engine.connect() as conn:
        yield conn

Test 1: the ledger cannot drift

Every movement of money is written as balanced entries, and no code path is allowed to write a single-sided one. The test is not a test at all in the usual sense; it is an assertion that runs after every test in the suite, so any new feature that touches the ledger inherits it for free.

@pytest.fixture(autouse=True)
def ledger_stays_balanced(db):
    yield
    unbalanced = db.execute(text('''
        SELECT transfer_id, currency, SUM(amount) AS delta
        FROM ledger_entries
        GROUP BY transfer_id, currency
        HAVING SUM(amount) <> 0
    ''')).all()
    assert not unbalanced, f'unbalanced transfers: {unbalanced}'

Two details carry most of the value. The grouping includes currency, because the first cross-currency feature is where a naive invariant quietly stops meaning anything. And amount is an exact numeric type, never a float. A float column is an automatic rejection at review; the invariant above would fail on rounding noise long before a human noticed the balances were wrong, which is the good case, and would pass while being subtly wrong in the bad one.

This fixture catches a specific and common mistake: a refund, chargeback, or fee handler that credits one account and forgets the counter-entry because the happy-path test only checked the user's visible balance.

Test 2: the same event twice

Providers retry. Queues redeliver. Operators click twice. Any endpoint that moves money has to be safe on the second delivery, and the only correct way to enforce that is a unique constraint on the external event identity plus a handler that treats the violation as success.

def test_webhook_is_idempotent(client, db):
    payload = provider_callback(external_id='pay_1', amount='100.00', status='captured')

    first = client.post('/webhooks/provider', json=payload)
    second = client.post('/webhooks/provider', json=payload)

    assert first.status_code == 200
    assert second.status_code == 200
    assert entry_count(db, external_id='pay_1') == 2  # one balanced pair, not two

What the gate rejects here is the pre-check: SELECT to see whether the event was already processed, then INSERT if not. It passes this test and fails in production, because the two deliveries arrive concurrently and both read an empty table. The constraint has to be in the schema, and the handler has to catch the integrity error rather than avoid it. A test that only sends the duplicate sequentially does not prove that, which is why the next test exists.

Test 3: two workers, one payout

Outbound money is a queue, and the interesting question is what happens when the queue is consumed twice at once, whether because of a scaling event, a redeploy overlap, or a stuck job being requeued while the original worker is still alive.

from concurrent.futures import ThreadPoolExecutor


def claim_payout(engine, payout_id):
    with engine.begin() as conn:
        claimed = conn.execute(text('''
            SELECT id FROM payout_jobs
            WHERE id = :id AND state = 'pending'
            FOR UPDATE SKIP LOCKED
        '''), {'id': payout_id}).first()
        if claimed is None:
            return False
        conn.execute(text('''
            UPDATE payout_jobs SET state = 'sent' WHERE id = :id
        '''), {'id': payout_id})
        return True


def test_only_one_worker_claims_a_payout(engine, pending_payout):
    with ThreadPoolExecutor(max_workers=2) as pool:
        results = list(pool.map(lambda _: claim_payout(engine, pending_payout.id), range(2)))

    assert sorted(results) == [False, True]

The test needs two real connections, because that is the whole point: a single-session test cannot observe a lock. SKIP LOCKED rather than a plain FOR UPDATE is deliberate. With a plain lock the second worker waits, then reads the row after the state change and has to re-check the state anyway; with SKIP LOCKED the loser is told immediately and moves to the next job, which is what keeps a payout batch flowing when one row is slow.

Test 4: reconciliation classifies, it does not heal

The fourth test is the one teams skip, and it is the one that pays for itself. Every money system eventually disagrees with its counterparty: a deposit arrives with a reference that matches no open invoice, an amount comes back short by the provider's fee, a capture is reported that we have no record of requesting. The reconciliation job's job is to name the disagreement, not to fix it.

@pytest.mark.parametrize('statement_row, expected_kind', [
    (statement(external_id='pay_1', amount='100.00'), 'matched'),
    (statement(external_id='pay_1', amount='99.00'), 'amount_mismatch'),
    (statement(external_id='pay_unknown', amount='50.00'), 'unattributed'),
    (statement(external_id='pay_1', amount='100.00', duplicate=True), 'duplicate'),
])
def test_reconciliation_classifies_instead_of_healing(db, captured_payment, statement_row, expected_kind):
    before = entry_count(db)

    report = reconcile(db, rows=[statement_row])

    assert report.single().kind == expected_kind
    assert entry_count(db) == before

The last assertion is the important one. Reconciliation writes discrepancy records and nothing else; it never posts a correcting entry on its own. A self-healing reconciler is the worst possible component to own a money path, because it turns a visible mismatch into an invisible adjustment, and the next mismatch is then reconciled against an already-adjusted state. When a case needs correcting, a human decides and the correction goes through the same ledger code as every other movement, with the same invariant applied.

The unattributed case deserves its own fixture rather than being folded into the mismatch bucket. Attribution by payment requisites is where inbound money actually goes wrong in production: the sender edits the reference, or pays from a different account than the one on file. Classifying that separately is what lets a compliance or finance operator work a queue instead of reading raw statements.

What it costs to run

Honestly: it is slower than a mocked suite. The CI job starts a database container, migrations run before the session, and the concurrency tests spend real time waiting on real locks. Money-path tests run in minutes rather than seconds, and we parallelise them with a schema per worker so the truncation fixture stays isolated.

One rule holds the whole thing together: we never quarantine a flaky money test. A nondeterministic result in this suite is not test infrastructure being annoying, it is the race we were trying to find, surfacing on the cheap side of production. Every time we have chased one instead of retrying it, there was something real underneath.

The gate

A change that touches the money path does not pass review with a float column for an amount, a mocked database, an idempotency pre-check instead of a constraint, a single-session concurrency test, or a reconciler that writes ledger entries. None of these are exotic engineering. They are the four questions we have learned to ask before the code goes anywhere near a real account, because it is the only stage where the answers are still cheap.

Originally published on shipmindlabs.com — where we write about payment systems, infrastructure and marketplace backends.

One payment protocol for every provider: hold, capture, refund

Shipmind Labs — Sat, 15 Aug 2026 17:56:59 +0000

A payment integration usually starts with one provider and ends with three, and by then the provider's vocabulary has leaked into the order code: one branch for the gateway that separates authorization from capture, another for the one that only does a single-shot charge, a third for the rail that settles asynchronously. This is the interface we put between our domain and any provider — four verbs, an idempotency key we generate ourselves, and an audit log that stores every request and every callback before anything is parsed. We are extracting this interface into an open-source package — github.com/shipmindlabs/payment-gateways — where the protocol and its value types landed first; the audit and callback pieces described below are being added in the open.

The constraint: order code outlives the provider

We have built payment services on card rails, on account-to-account flows, and on stablecoin rails, and the thing that repeats across all of them is that the provider changes long before the business logic does. A contract lapses, a second rail gets added for another market, a gateway starts declining a whole category of transactions. If the checkout flow, the order state machine and the payout job each know which provider they are talking to, that swap turns into a rewrite of code that has nothing to do with payments.

So the abstraction has a narrow goal. It is not there to make providers look identical, because they are not. It is there to make the differences explicit, enumerated, and confined to one directory.

Four verbs and a capability set

Every provider we have integrated fits into authorize, capture, refund and cancel, plus one function that turns an inbound HTTP request into a normalized event. What differs is which verbs are real and which are emulated, and that belongs in data rather than in an if somewhere in the order service.

from dataclasses import dataclass
from decimal import Decimal
from enum import Enum
from typing import Protocol


class IntentState(str, Enum):
    PENDING = 'pending'
    AUTHORIZED = 'authorized'
    CAPTURED = 'captured'
    REFUNDED = 'refunded'
    CANCELED = 'canceled'
    FAILED = 'failed'


@dataclass(frozen=True)
class Money:
    amount: Decimal
    currency: str


@dataclass(frozen=True)
class GatewayResult:
    state: IntentState
    provider_reference: str | None
    exchange_id: int          # row in the audit log that produced this result
    decline_reason: str | None = None


class PaymentGateway(Protocol):
    name: str
    capabilities: frozenset[str]   # 'hold', 'partial_capture', 'partial_refund'

    def authorize(self, intent_id: str, amount: Money, key: str) -> GatewayResult: ...
    def capture(self, intent_id: str, amount: Money, key: str) -> GatewayResult: ...
    def refund(self, intent_id: str, amount: Money, key: str) -> GatewayResult: ...
    def cancel(self, intent_id: str, key: str) -> GatewayResult: ...
    def parse_callback(self, body: bytes, headers: dict) -> 'CallbackEvent': ...

The result type is deliberately poor. It carries our state, the provider's reference, and the id of the audit row it came from — not the provider's status string, not its error codes. Anything richer starts pulling provider concepts back up into the caller.

The idempotency key must be a function of the intent

The key exists so that a retry is not a second charge. That only works if the retry computes the same key as the original attempt, which rules out generating a fresh UUID at call time: the process that times out and restarts will happily produce a new key and a second authorization.

We derive the key from what the operation is about — the intent, the verb, and a sequence number for operations that can legitimately repeat, such as partial captures.

import hashlib


def idempotency_key(intent_id: str, verb: str, sequence: int = 0) -> str:
    material = f'{verb}:{intent_id}:{sequence}'
    return hashlib.sha256(material.encode()).hexdigest()[:40]

The key is written to our database in the same transaction that creates the operation row, before the HTTP call goes out. A crashed worker picks the row back up, recomputes the identical key, and the provider answers with the original result instead of taking the money again. Providers that ignore idempotency headers get the same treatment on our side: the operation row has a unique constraint on the key, so we cannot even start a duplicate attempt without a conflict.

Write the exchange down before you understand it

Every outbound request, every response, and every inbound callback goes into one table, stored raw, before any parsing happens.

create table gateway_exchange (
    id                bigserial primary key,
    provider          text        not null,
    direction         text        not null check (direction in ('request', 'response', 'callback')),
    intent_id         uuid,
    idempotency_key   text,
    external_event_id text,
    payload           jsonb       not null,
    headers           jsonb       not null default '{}'::jsonb,
    created_at        timestamptz not null default now()
);

create unique index gateway_exchange_callback_uniq
    on gateway_exchange (provider, external_event_id)
    where direction = 'callback';

The ordering matters more than the schema. A callback handler that parses first and stores second loses exactly the payloads that are most worth having: the ones with a field the provider added last week, the ones that made the parser throw. Store, commit, then interpret. When a dispute arrives months later, or when a provider insists it sent a notification, the answer is a query rather than a recollection.

Card data must never reach this table, which is an argument for hosted payment pages wherever the rail offers them: what we store is then references and status, not instruments. What does reach the table is authentication material in headers, and that gets redacted on the way in — the log is evidence, not a secret store.

What a retried webhook does to a naive handler

The naive handler reads the event type, marks the order paid, and credits a balance. Then the provider retries, because our response was slow, or because a proxy returned 502 after our transaction had already committed. Nothing about the retry is exceptional — it is the provider behaving correctly — and the balance is credited twice.

Two defenses, both cheap. The first is the partial unique index above: the insert of the callback row is the deduplication.

def handle_callback(provider: str, body: bytes, headers: dict) -> int:
    exchange_id = store_exchange(provider, body, headers)   # ON CONFLICT DO NOTHING
    if exchange_id is None:
        return 200                                          # already seen, nothing to do
    event = REGISTRY[provider].parse_callback(body, headers)
    apply_event(event, exchange_id)
    return 200

The second is that state changes are compare-and-set against the expected state, not blind writes.

update payment_intent
   set state = $2, updated_at = now()
 where id = $1
   and state = any($3)      -- states from which $2 is reachable
returning state;

Zero rows updated is a normal outcome, not an error: it means the intent has already moved on, either because a poller got there first or because this is a duplicate that slipped past deduplication. The transition table is small enough to read in one screen — pending goes to authorized, canceled or failed; authorized goes to captured, canceled or failed; captured goes to refunded; the terminal states go nowhere.

One more rule that is easy to get backwards: return 5xx only when we genuinely failed to record the event. The status code is the only lever the provider has, and a handler that returns 500 on a duplicate teaches the provider to hammer it.

Adding the second gateway

With the protocol in place, a second provider is an adapter, a row in the registry, and a capability set. The order service keeps calling the same four methods, and the one place that has to care about differences is the point where a capability is missing.

gateway = REGISTRY[intent.provider]
if 'hold' in gateway.capabilities:
    gateway.authorize(intent.id, amount, idempotency_key(intent.id, 'authorize'))
else:
    intent.defer_charge_until_fulfilment()

Non-card rails fit the same four verbs with different physics. An account-to-account transfer authorizes and captures in one motion, and its refund is a fresh outbound payment rather than a mutation of the original — so refund returns a new intent reference, and the caller was written to expect that from the beginning. A stablecoin transfer is the same shape again, with confirmation depth standing in for the settlement delay. None of this is hidden; it is the reason the capability set exists.

What it costs to run

The audit log becomes the largest table in the payment service. Partition it by month, decide the retention window against compliance requirements rather than disk comfort, and index only the fields you actually query — provider, intent, external event id — instead of a blanket index over the payload.

Holds expire. An authorization has a lifetime, and if fulfilment happens after it lapses, capture fails at the worst possible moment. That needs a sweeper that re-authorizes or cancels ahead of expiry, and it needs to be visible, because its failures are silent by nature.

Callbacks go missing. Some fraction of events never arrive, so a reconciliation job polls the provider's status endpoint for intents that have been pending longer than expected. It writes into the same audit log and applies events through the same compare-and-set path, which means the recovery route is not a second, less-tested implementation of the first.

And because every exchange is recorded, adapter tests get their fixtures for free: replay the stored transcripts against the adapter and assert the normalized result. When a provider changes a payload shape, the transcript that broke production becomes the test that stops it happening twice.

The protocol is not sophisticated, and that is most of the point. Four verbs, a key derived from the intent, a table that remembers everything, and state transitions that refuse to run backwards — after that, a new provider is a week of adapter work instead of a quarter of untangling.

Catching money bugs with ledger invariants, not error logs

Shipmind Labs — Mon, 10 Aug 2026 17:55:43 +0000

A payout that credits the wrong sub-account returns HTTP 200. Nothing throws, the worker acknowledges the message, the error dashboards stay green, and the discrepancy surfaces days later when someone reconciles a bank statement by hand.

That gap — between "the code ran without errors" and "the money is where it should be" — is where the hardest incidents in payment systems live. Retries, partial failures, and provider callbacks arriving out of order all produce states that are individually plausible and collectively wrong. Exception tracking cannot see them, because there is no exception.

We work on systems that move money: payment services on card rails and account-to-account flows, virtual bank-account ledgering with per-user attribution, custodial wallets, marketplace payout flows. Across all of them the same practice earns its keep. Write down the properties that must hold over the ledger, check them continuously, and treat a violation with the same weight as a spike of 500s.

The constraint that makes this hard

A ledger backed by external rails is legitimately inconsistent for a while. A transfer is submitted, the provider confirms asynchronously, the bank statement line arrives on the next value date. A check that ignores that timing produces false positives, and a team that gets paged for false positives stops reading pages. So an invariant is not just a boolean over the tables — it needs a settlement window and a severity.

The second constraint is operational: these are the tables you cannot lock or slow down. Checks must be read-only, run against a replica, and be cheap enough to repeat every few minutes.

Write the invariants as SQL, not as prose

We keep them in the repository next to the migrations, one file per invariant, under version control and code review like everything else. The contract is deliberately narrow: each query returns zero rows when the system is healthy, and one row per offending entity when it is not. Every file takes a single :settlement parameter cast to an interval.

Double entry, first. Every posted transaction must sum to zero:

-- ledger/invariants/sql/0001_transaction_balances_to_zero.sql
SELECT t.id AS entity_id,
       SUM(p.amount_minor) AS residual_minor
FROM ledger_transaction t
JOIN ledger_posting p ON p.transaction_id = t.id
WHERE t.posted_at < now() - CAST(:settlement AS interval)
GROUP BY t.id
HAVING SUM(p.amount_minor) <> 0;

Next, the cached balance every read path uses must match the postings that produced it. This is the invariant that catches a balance updated outside a transaction, or updated twice by a retry:

-- ledger/invariants/sql/0002_account_balance_matches_postings.sql
SELECT a.id AS entity_id,
       a.balance_minor AS cached_minor,
       COALESCE(SUM(p.amount_minor), 0) AS derived_minor
FROM ledger_account a
LEFT JOIN ledger_posting p ON p.account_id = a.id
WHERE a.updated_at < now() - CAST(:settlement AS interval)
GROUP BY a.id, a.balance_minor
HAVING a.balance_minor <> COALESCE(SUM(p.amount_minor), 0);

Then attribution. In a virtual-account design, where each user pays into their own requisites and the bank reports movements on the parent account, incoming money has to be matched to exactly one user. Zero matches means funds are sitting unattributed; two matches means one payment was credited twice:

-- ledger/invariants/sql/0003_inbound_lines_attributed_once.sql
SELECT l.id AS entity_id,
       count(p.id) AS attribution_count
FROM bank_statement_line l
LEFT JOIN ledger_posting p ON p.statement_line_id = l.id
WHERE l.direction = 'inbound'
  AND l.value_date < current_date - CAST(:settlement AS interval)
GROUP BY l.id
HAVING count(p.id) <> 1;

The rest of the set follows the same shape: no custody or escrow account below zero at any moment, no posting written to a closed account, the sum of user balances equal to the omnibus balance reported by the provider, no transaction in a non-terminal state older than its expected lifetime.

Two conventions do a lot of work here. Amounts are integers in minor units, never floats, so equality comparisons are meaningful. And every invariant returns an entity_id, so an alert points at rows an engineer can open rather than at a number that dropped.

Run them like tests, on a schedule

The runner is intentionally dull. It loads the SQL, executes it read-only, counts rows, samples a few for the alert payload, and reports how long it took.

# ledger/invariants/runner.py
import time
from dataclasses import dataclass
from pathlib import Path

from sqlalchemy import text

SQL_DIR = Path(__file__).parent / "sql"


@dataclass(frozen=True)
class Invariant:
    name: str
    severity: str        # "page" or "ticket"
    settlement: str      # interval literal, e.g. "5 minutes"
    sample_size: int = 20


REGISTRY = (
    Invariant("0001_transaction_balances_to_zero", "page", "1 minute"),
    Invariant("0002_account_balance_matches_postings", "page", "5 minutes"),
    Invariant("0003_inbound_lines_attributed_once", "ticket", "2 days"),
    Invariant("0004_custody_never_negative", "page", "0 seconds"),
)


def run(invariant, session):
    sql = (SQL_DIR / f"{invariant.name}.sql").read_text()
    started = time.monotonic()
    rows = session.execute(
        text(sql), {"settlement": invariant.settlement}
    ).mappings().all()
    return {
        "name": invariant.name,
        "severity": invariant.severity,
        "violations": len(rows),
        "sample": [dict(r) for r in rows[: invariant.sample_size]],
        "duration_s": time.monotonic() - started,
    }

The scheduled task wraps it, pins the session to a read-only transaction on the replica, and publishes one gauge per invariant:

# ledger/invariants/tasks.py
from sqlalchemy import text

from app.celery import app
from app.db import replica_session
from app.metrics import gauge
from app.alerts import emit
from ledger.invariants.runner import REGISTRY, run


@app.task(name="ledger.run_invariants")
def run_invariants():
    with replica_session() as session:
        session.execute(text("SET TRANSACTION READ ONLY"))
        for invariant in REGISTRY:
            result = run(invariant, session)
            tags = [f"invariant:{invariant.name}"]
            gauge("ledger.invariant.violations", result["violations"], tags)
            gauge("ledger.invariant.duration_s", result["duration_s"], tags)
            gauge("ledger.invariant.last_run_ts", time.time(), tags)
            if result["violations"]:
                emit(result)

The third gauge matters more than it looks. A checker that stopped running produces exactly the same picture as a perfectly healthy ledger: no violations, no alerts. So the monitoring has two rules per invariant — one on the violation count, and one on the age of last_run_ts. Without the staleness rule, the whole mechanism can fail silently, which is the failure mode it exists to prevent.

The same invariants in the test suite

Because the checks are plain SQL against the schema, the test suite can run them too — with a zero settlement window, since a test controls time and has no in-flight provider callbacks. Our money-path tests end with the ledger being asserted clean, not just with the response body being asserted correct:

# tests/conftest.py
import pytest

from ledger.invariants.runner import REGISTRY, run


@pytest.fixture
def assert_ledger_clean(db_session):
    def _assert(exclude=()):
        failures = []
        for invariant in REGISTRY:
            if invariant.name in exclude:
                continue
            result = run(invariant, db_session)
            if result["violations"]:
                failures.append((invariant.name, result["sample"]))
        assert not failures, failures

    return _assert

# tests/payouts/test_reversal.py
def test_failed_payout_reverses_without_residual(
    api_client, payout_factory, provider_stub, assert_ledger_clean
):
    payout = payout_factory(state="submitted")
    provider_stub.reject(payout.reference, reason="account_closed")

    api_client.post("/internal/payouts/poll")
    api_client.post("/internal/payouts/poll")  # duplicate poll, must be idempotent

    assert_ledger_clean()

This is also what our review gate looks for. A pull request that introduces a new money path — a new transfer type, a new reversal branch, a new provider — is expected either to be covered by an existing invariant or to bring a new one with it. Reviewers ask the same question every time: which property of the ledger would be violated if this code is wrong, and what checks it? Answering that in review costs minutes. Answering it during a reconciliation incident costs a week and a lot of trust.

What this costs to run

The frequent invariants are aggregates over the postings table, so they live or die on indexes: ledger_posting(transaction_id), ledger_posting(account_id), and a timestamp index supporting the settlement filter. Past a certain table size a full sweep every few minutes stops being free, and the fix is to split the cadence:

Mode	Cadence	Scope
Incremental	minutes	entities touched since the last watermark
Full sweep	nightly, off-peak	the whole table set

Incremental mode adds one predicate to the same file, driven by a stored watermark:

AND t.posted_at > CAST(:since AS timestamptz)

The full sweep still matters, because incremental checks only see rows the application touched. Corruption introduced by a manual fix, a bad backfill, or a restore lands outside the watermark, and only the sweep finds it.

Operationally it is a replica, one scheduled task, a handful of gauges, and two alert rules per invariant. No new infrastructure, no vendor.

Close

Error rates tell you whether the code ran. Invariants tell you whether the money is right, and those are different questions. Since the ledger is the source of truth for every balance a user sees and every figure that ends up in a reconciliation report, it should also be the thing under continuous assertion — in production and in the test suite, with the same SQL doing both jobs.