One payment protocol for every provider: hold, capture, refund

A payment integration usually starts with one provider and ends with three, and by then the provider's vocabulary has leaked into the order code: one branch for the gateway that separates authorization from capture, another for the one that only does a single-shot charge, a third for the rail that settles asynchronously. This is the interface we put between our domain and any provider — four verbs, an idempotency key we generate ourselves, and an audit log that stores every request and every callback before anything is parsed. We are extracting this interface into an open-source package — github.com/shipmindlabs/payment-gateways — where the protocol and its value types landed first; the audit and callback pieces described below are being added in the open.

The constraint: order code outlives the provider

We have built payment services on card rails, on account-to-account flows, and on stablecoin rails, and the thing that repeats across all of them is that the provider changes long before the business logic does. A contract lapses, a second rail gets added for another market, a gateway starts declining a whole category of transactions. If the checkout flow, the order state machine and the payout job each know which provider they are talking to, that swap turns into a rewrite of code that has nothing to do with payments.

So the abstraction has a narrow goal. It is not there to make providers look identical, because they are not. It is there to make the differences explicit, enumerated, and confined to one directory.

Four verbs and a capability set

Every provider we have integrated fits into authorize, capture, refund and cancel, plus one function that turns an inbound HTTP request into a normalized event. What differs is which verbs are real and which are emulated, and that belongs in data rather than in an if somewhere in the order service.

from dataclasses import dataclass
from decimal import Decimal
from enum import Enum
from typing import Protocol


class IntentState(str, Enum):
    PENDING = 'pending'
    AUTHORIZED = 'authorized'
    CAPTURED = 'captured'
    REFUNDED = 'refunded'
    CANCELED = 'canceled'
    FAILED = 'failed'


@dataclass(frozen=True)
class Money:
    amount: Decimal
    currency: str


@dataclass(frozen=True)
class GatewayResult:
    state: IntentState
    provider_reference: str | None
    exchange_id: int          # row in the audit log that produced this result
    decline_reason: str | None = None


class PaymentGateway(Protocol):
    name: str
    capabilities: frozenset[str]   # 'hold', 'partial_capture', 'partial_refund'

    def authorize(self, intent_id: str, amount: Money, key: str) -> GatewayResult: ...
    def capture(self, intent_id: str, amount: Money, key: str) -> GatewayResult: ...
    def refund(self, intent_id: str, amount: Money, key: str) -> GatewayResult: ...
    def cancel(self, intent_id: str, key: str) -> GatewayResult: ...
    def parse_callback(self, body: bytes, headers: dict) -> 'CallbackEvent': ...

The result type is deliberately poor. It carries our state, the provider's reference, and the id of the audit row it came from — not the provider's status string, not its error codes. Anything richer starts pulling provider concepts back up into the caller.

The idempotency key must be a function of the intent

The key exists so that a retry is not a second charge. That only works if the retry computes the same key as the original attempt, which rules out generating a fresh UUID at call time: the process that times out and restarts will happily produce a new key and a second authorization.

We derive the key from what the operation is about — the intent, the verb, and a sequence number for operations that can legitimately repeat, such as partial captures.

import hashlib


def idempotency_key(intent_id: str, verb: str, sequence: int = 0) -> str:
    material = f'{verb}:{intent_id}:{sequence}'
    return hashlib.sha256(material.encode()).hexdigest()[:40]

The key is written to our database in the same transaction that creates the operation row, before the HTTP call goes out. A crashed worker picks the row back up, recomputes the identical key, and the provider answers with the original result instead of taking the money again. Providers that ignore idempotency headers get the same treatment on our side: the operation row has a unique constraint on the key, so we cannot even start a duplicate attempt without a conflict.

Write the exchange down before you understand it

Every outbound request, every response, and every inbound callback goes into one table, stored raw, before any parsing happens.

create table gateway_exchange (
    id                bigserial primary key,
    provider          text        not null,
    direction         text        not null check (direction in ('request', 'response', 'callback')),
    intent_id         uuid,
    idempotency_key   text,
    external_event_id text,
    payload           jsonb       not null,
    headers           jsonb       not null default '{}'::jsonb,
    created_at        timestamptz not null default now()
);

create unique index gateway_exchange_callback_uniq
    on gateway_exchange (provider, external_event_id)
    where direction = 'callback';

The ordering matters more than the schema. A callback handler that parses first and stores second loses exactly the payloads that are most worth having: the ones with a field the provider added last week, the ones that made the parser throw. Store, commit, then interpret. When a dispute arrives months later, or when a provider insists it sent a notification, the answer is a query rather than a recollection.

Card data must never reach this table, which is an argument for hosted payment pages wherever the rail offers them: what we store is then references and status, not instruments. What does reach the table is authentication material in headers, and that gets redacted on the way in — the log is evidence, not a secret store.

What a retried webhook does to a naive handler

The naive handler reads the event type, marks the order paid, and credits a balance. Then the provider retries, because our response was slow, or because a proxy returned 502 after our transaction had already committed. Nothing about the retry is exceptional — it is the provider behaving correctly — and the balance is credited twice.

Two defenses, both cheap. The first is the partial unique index above: the insert of the callback row is the deduplication.

def handle_callback(provider: str, body: bytes, headers: dict) -> int:
    exchange_id = store_exchange(provider, body, headers)   # ON CONFLICT DO NOTHING
    if exchange_id is None:
        return 200                                          # already seen, nothing to do
    event = REGISTRY[provider].parse_callback(body, headers)
    apply_event(event, exchange_id)
    return 200

The second is that state changes are compare-and-set against the expected state, not blind writes.

update payment_intent
   set state = $2, updated_at = now()
 where id = $1
   and state = any($3)      -- states from which $2 is reachable
returning state;

Zero rows updated is a normal outcome, not an error: it means the intent has already moved on, either because a poller got there first or because this is a duplicate that slipped past deduplication. The transition table is small enough to read in one screen — pending goes to authorized, canceled or failed; authorized goes to captured, canceled or failed; captured goes to refunded; the terminal states go nowhere.

One more rule that is easy to get backwards: return 5xx only when we genuinely failed to record the event. The status code is the only lever the provider has, and a handler that returns 500 on a duplicate teaches the provider to hammer it.

Adding the second gateway

With the protocol in place, a second provider is an adapter, a row in the registry, and a capability set. The order service keeps calling the same four methods, and the one place that has to care about differences is the point where a capability is missing.

gateway = REGISTRY[intent.provider]
if 'hold' in gateway.capabilities:
    gateway.authorize(intent.id, amount, idempotency_key(intent.id, 'authorize'))
else:
    intent.defer_charge_until_fulfilment()

Non-card rails fit the same four verbs with different physics. An account-to-account transfer authorizes and captures in one motion, and its refund is a fresh outbound payment rather than a mutation of the original — so refund returns a new intent reference, and the caller was written to expect that from the beginning. A stablecoin transfer is the same shape again, with confirmation depth standing in for the settlement delay. None of this is hidden; it is the reason the capability set exists.

What it costs to run

The audit log becomes the largest table in the payment service. Partition it by month, decide the retention window against compliance requirements rather than disk comfort, and index only the fields you actually query — provider, intent, external event id — instead of a blanket index over the payload.

Holds expire. An authorization has a lifetime, and if fulfilment happens after it lapses, capture fails at the worst possible moment. That needs a sweeper that re-authorizes or cancels ahead of expiry, and it needs to be visible, because its failures are silent by nature.

Callbacks go missing. Some fraction of events never arrive, so a reconciliation job polls the provider's status endpoint for intents that have been pending longer than expected. It writes into the same audit log and applies events through the same compare-and-set path, which means the recovery route is not a second, less-tested implementation of the first.

And because every exchange is recorded, adapter tests get their fixtures for free: replay the stored transcripts against the adapter and assert the normalized result. When a provider changes a payload shape, the transcript that broke production becomes the test that stops it happening twice.

The protocol is not sophisticated, and that is most of the point. Four verbs, a key derived from the intent, a table that remembers everything, and state transitions that refuse to run backwards — after that, a new provider is a week of adapter work instead of a quarter of untangling.