In the previous post I took a payment platform apart by cost: three layers, where the person-years go, what you can avoid writing. It left one question honestly open: fine, but how do you actually assemble it?
This is the answer. A reference architecture: the ledger model, transaction boundaries, module split, cluster topology, integration routes, and recovery points. With code and diagrams.
Caveat up front — this is a reference model, not a dump from one production system. The code shows the real API and patterns that work, but your accounting model will differ, and it should. More on that at the end.
The stack: redb, a typed store on PostgreSQL; redb.Route, an integration engine; redb.Tsak, a runtime; and redb.Identity, an OIDC/OAuth 2.1 server. All Pro, all free across the 3.x line.
Part of the redb / redb.Route series — recent posts first:
- A payment platform on .NET: what it actually costs, and how much of it you never have to write
- redb 3.4.0: day-two operations for a .NET stack
- Leaving MassTransit for a Camel state of mind: the Kafka connector, Scatter-Gather, and transactions
- Apache Camel for .NET, dissected: the HTTP connector with no ASP.NET MVC + the Content-Based Router
- redb.Route — Apache Camel for .NET: 22 transports, 30+ EIP patterns, compiled DSL
Sources: github.com/redbase-app. Docs: redbase.app.
The whole thing on one screen
Top down first. Here's the platform in one picture; we'll take each block apart below.
OUTSIDE WORLD
browser/mobile acquirer bank partner regulator
│ │ │ │ │
HTTPS HTTPS IBM MQ SFTP exports
│ │ │ │ │
╔═══════▼══════════════▼═══════════▼══════════▼══════════▼═══════╗
║ TSAK WORKER (clustered) ║
║ ║
║ ┌──────────────┐ ┌───────────────────────────────────────┐ ║
║ │ identity │ │ payments │ ║
║ │ .tpkg │ │ │ ║
║ │ │ │ payments.Api ← HTTP facade │ ║
║ │ OAuth 2.1 │◄─┼─ payments.Acq ← acquirers │ ║
║ │ OIDC │ │ payments.Bank ← IBM MQ │ ║
║ │ SCIM │ │ payments.Files ← SFTP registries │ ║
║ │ audit │ │ payments.Recon ← reconciliation │ ║
║ │ │ │ payments.Core ← LEDGER + SCHEMAS │ ║
║ └──────┬───────┘ └───────────────┬───────────────────────┘ ║
║ │ direct-vm:// │ ║
║ │ (no network) │ ║
╚═════════╪══════════════════════════╪═══════════════════════════╝
│ │
┌────▼─────┐ ┌────▼────────┐
│ identity │ │ payments │
│ DB │ │ DB │
└──────────┘ └─────────────┘
separate PostgreSQL
named store (Pro)
Four things worth noticing immediately, because they drive everything else:
Identity runs in the same worker but on its own database. Payment modules reach it over direct-vm:// — an in-process transport, no socket and no TLS. From the outside it stays a normal OIDC server on HTTPS for browsers and mobile apps.
payments.Core has no external transport at all. It's pure ledger: schemas, posting rules, balance computation. Every entrance to it is direct-vm:// from a neighbouring module.
Each external protocol is its own module. The acquirer changed something and its module needs redeploying — the bank rail doesn't notice.
Many modules, one worker. Or several — but that's an operations decision, not an architectural one, and it's a config change. We'll come back to it in the cluster section.
The ledger: a posting that never changes
Start with the data model, because everything else follows from it — transaction boundaries, idempotency, reconciliation, and what you'll be able to tell a regulator three years from now.
Decision #1: postings are append-only
The single most important architectural decision fits in one sentence: a posting object is created once and never modified. No status field, no cancellation, no amount correction. Made a mistake? Write a reversing posting; don't edit the old one.
Why this matters specifically here: redb has no built-in object versioning. If you update postings, you'll be building the change history yourself. If you don't — the history is the posting journal, and there's nothing to build.
That's not an engine limitation, that's correct accounting. Bookkeeping has worked this way for two hundred years.
[RedbScheme(Name = "payments.posting", Alias = "Posting")]
public class PostingProps
{
/// <summary>Business operation key. One operation → several postings sharing it.</summary>
public string OperationId { get; set; } = "";
/// <summary>Account. A flat id, not a reference — see below.</summary>
public long AccountId { get; set; }
/// <summary>Signed delta: positive is a credit, negative is a debit.</summary>
public decimal Delta { get; set; }
public RedbListItem? Currency { get; set; }
public RedbListItem? Kind { get; set; }
public DateTimeOffset OccurredAt { get; set; }
/// <summary>Set only on reversals — points at the posting being reversed.</summary>
public long? ReversesPostingId { get; set; }
public string? Reference { get; set; }
}
Three details, each a deliberate choice.
decimal Delta lands in a NUMERIC(38,18) column. Nothing to configure: any decimal in props maps to a column with 38 digits of precision, 18 of them after the point. Fees, FX and tax compute without accumulating rounding error — and eighteen decimals happens to be exactly the precision Ether is denominated in, if a crypto line shows up next year.
long AccountId instead of RedbObject<AccountProps>. redb supports object references natively, and for domain entities they're great — the whole graph loads in one call. But postings are read in the millions and always by account, so what's needed here is a fast scalar filter, not a graph. A flat long hits a partial index on numeric values and behaves like an ordinary column. The rule: use a graph where the object is read whole; use a flat key where you filter by it.
RedbListItem for currency and operation kind. These are redb's built-in lookup lists: the value is stored as a reference to a list item, and you can filter both by item identity and by its string value. No enum tables and no joins.
Decision #2: the account has no balance field
[RedbScheme(Name = "payments.account", Alias = "Account")]
public class AccountProps
{
public string Number { get; set; } = "";
public long OwnerId { get; set; }
public RedbListItem? Currency { get; set; }
public RedbListItem? AccountType { get; set; }
public DateTimeOffset OpenedAt { get; set; }
public DateTimeOffset? ClosedAt { get; set; }
// There is NO Balance field here. On purpose.
}
A balance field on the account is the most common and most expensive mistake in payment systems. It immediately creates two sources of truth: the balance in the field, and the balance as the sum of postings. They diverge. Always. The only question is when you notice.
A balance is a function of the journal, full stop:
var balance = await redb.Query<PostingProps>()
.Where(p => p.AccountId == accountId)
.SumAsync(p => p.Delta);
One query, aggregated database-side, nothing pulled into memory.
Decision #3: the snapshot is an optimisation, not the truth
On an account with a million postings you obviously can't sum everything every time. Hence snapshots:
[RedbScheme(Name = "payments.balance_snapshot", Alias = "Balance snapshot")]
public class BalanceSnapshotProps
{
public long AccountId { get; set; }
public decimal Amount { get; set; }
/// <summary>This snapshot accounts for every posting with id ≤ this value.</summary>
public long UpToPostingId { get; set; }
public DateTimeOffset TakenAt { get; set; }
}
The balance is then "latest snapshot plus deltas since":
var snap = await redb.Query<BalanceSnapshotProps>()
.Where(s => s.AccountId == accountId)
.OrderByDescending(s => s.UpToPostingId)
.FirstOrDefaultAsync();
var baseAmount = snap?.Props.Amount ?? 0m;
var fromId = snap?.Props.UpToPostingId ?? 0L;
var delta = await redb.Query<PostingProps>()
.Where(p => p.AccountId == accountId)
.WhereRedb(o => o.Id > fromId)
.SumAsync(p => p.Delta);
var balance = baseAmount + delta;
The key property: a snapshot can be thrown away and recomputed at any moment, because the source of truth is the journal. Snapshot corrupted, stale, or computed by old logic? Delete it and take a new one. With a balance field on the account you don't get that luxury — once it diverges, you no longer know which number is right.
Snapshots are taken by a scheduled background job — an ordinary route, covered in the integrations section.
Decision #4: idempotency belongs in the model, not in a side table
The operation key lives inside the posting, not in a separate "processed messages" table. Which makes "has this operation already been posted?" an ordinary query:
var alreadyPosted = await redb.Query<PostingProps>()
.Where(p => p.OperationId == operationId)
.AnyAsync();
Why this beats a side table: it cannot get out of sync. The posting and the record that it happened are the same object, written by the same operation. The classic two-table shape permits "marker exists, posting doesn't" and vice versa — and that gets cleaned up by hand.
A route-level idempotent consumer still exists, but it solves a different problem: rejecting redelivery before we ever reach the database. Two different lines of defence; neither replaces the other.
The model as a whole
┌────────────────┐ ┌─────────────────────┐
│ Account │ │ BalanceSnapshot │
│ payments. │◄────────┤ payments. │
│ account │ 1:N │ balance_snapshot │
│ │ │ │
│ Number │ │ AccountId │
│ OwnerId │ │ Amount │
│ Currency │ │ UpToPostingId ─────┼──┐
│ AccountType │ │ TakenAt │ │
│ (no balance!) │ └─────────────────────┘ │
└───────┬────────┘ │
│ 1:N │
│ │
┌───────▼───────────────────────────────┐ │
│ Posting │ │
│ payments.posting │◄──────────┘
│ │ snapshot "up to"
│ OperationId ← idempotency │
│ AccountId │
│ Delta ← NUMERIC(38,18) │
│ Currency ← RedbListItem │
│ Kind ← RedbListItem │
│ OccurredAt │
│ ReversesPostingId ──┐ │
│ Reference │ reversal │
└──────────────────────┴────────────────┘
APPEND-ONLY. Never updated.
Notice what the model doesn't have: an idempotency table, a change-history table, a status field on the posting, a balance field on the account. Each is absent for its own reason, and every reason is the same underlying one — don't create a second source of truth.
Transaction boundaries: where exactly the line runs
The previous section was about what to store. This one is about what has to happen atomically. This is where people get it wrong most often, and where being wrong costs the most.
There's one rule and it's rigid:
INSIDE ONE TRANSACTION OUTSIDE — NEVER
────────────────────── ───────────────
✓ every posting of one operation ✗ HTTP to the acquirer
✓ the outbox row ✗ publishing to a broker
✓ the balance snapshot update ✗ sending an email
✓ the idempotency marker ✗ any network call
(which is the posting itself) ✗ waiting on anyone else
The reason is grade-school simple: a transaction holds locks, and a network call can take thirty seconds. An external call inside a transaction is guaranteed degradation under load plus entertaining deadlocks at 3 a.m.
Which is also why the outbox isn't a luxury but a necessity: you cannot publish to a broker atomically with a posting, but you can write to your own database.
What it looks like in code
public async Task<TransferResult> TransferAsync(
long fromAccountId, long toAccountId, decimal amount,
string operationId, CancellationToken ct)
{
// 1. Cheap rejection BEFORE the transaction: shed retries without taking locks
if (await redb.Query<PostingProps>()
.Where(p => p.OperationId == operationId).AnyAsync())
return TransferResult.AlreadyProcessed;
await using var tx = await redb.Context.BeginTransactionAsync();
// 2. Lock accounts STRICTLY IN ASCENDING id ORDER.
// Otherwise two opposing transfers A→B and B→A deadlock.
var locked = new[] { fromAccountId, toAccountId };
Array.Sort(locked);
await redb.LockForUpdateAsync(locked);
// 3. Re-check — now under the lock (TOCTOU guard)
if (await redb.Query<PostingProps>()
.Where(p => p.OperationId == operationId).AnyAsync())
return TransferResult.AlreadyProcessed; // dispose rolls back
// 4. Sufficient funds — evaluated under that same lock
if (await GetBalanceAsync(fromAccountId) < amount)
return TransferResult.InsufficientFunds;
var now = DateTimeOffset.UtcNow;
// 5. Both postings in one batch, one round-trip
await redb.AddNewObjectsAsync(new[]
{
Posting(operationId, fromAccountId, -amount, now),
Posting(operationId, toAccountId, +amount, now),
});
// 6. The event goes to the outbox IN THE SAME TRANSACTION
await redb.SaveAsync(OutboxEvent("transfer.completed", operationId, now));
await tx.CommitAsync();
return TransferResult.Posted;
}
The non-obvious parts.
The double idempotency check isn't paranoia. The first, outside the transaction, sheds mass retries cheaply: it takes no locks and blocks nobody. The second, under the lock, closes the window between check and write. Without the first the system degrades on a retry storm; without the second it lets a double posting through.
Sorting ids before locking is mandatory. Textbook, and routinely forgotten. Two opposing transfers — A→B and B→A — lock the accounts in opposite order and wedge. A single canonical acquisition order removes an entire class of deadlocks; the store itself uses the same trick internally for batch operations.
The balance check sits under the same lock as the write. Check without the lock and you get the classic race that produces a double debit.
Both postings go in one batch. AddNewObjectsAsync is a single round-trip rather than two. On a transfer the difference is small; on bulk charging it's decisive.
If you don't need explicit rollback control, there's a shorter form:
await redb.Context.ExecuteAtomicAsync(async () =>
{
await redb.AddNewObjectsAsync(postings);
await redb.SaveAsync(outboxEvent);
});
Why this matters for sharding later
Notice that the entire transaction fits inside one database. That isn't an accident — it's a requirement you have to design in from day one.
There is no distributed transaction across shards in cross-platform .NET; two-phase commit over two PostgreSQL instances isn't available. So if the platform ever grows sideways, the shard key must be chosen so both sides of a transfer land on the same shard. In practice that means sharding by customer or by account group, not by payment id.
You make this decision once, at the start. Redoing it on live data is a separate project.
The boundary, drawn
HTTP request
│
▼
┌──────────────────────────────────────────────┐
│ 1. Idempotency check (no locks taken) │ ← outside the transaction
└───────────────────┬──────────────────────────┘
│
╔═══════════════════▼═══════════════════════════╗
║ TRANSACTION ║
║ ║
║ 2. LockForUpdate(accounts, ascending id) ║
║ 3. Idempotency check, again ║
║ 4. Sufficient-funds check ║
║ 5. AddNewObjects(postings) ║
║ 6. Save(outbox event) ║
║ ║
║ COMMIT ────────────────────────────┐ ║
╚═══════════════════════════════════════════╪═══╝
│
┌───────────────────────┘
│ from here on: async, outside the transaction
▼
┌───────────────────────────────────────────────┐
│ Sql.Poll(outbox) → Kafka → acquirer / bank │
│ Wait on the network as long as you like: │
│ locks are released, money is already posted │
└───────────────────────────────────────────────┘
Module split: unit of deploy = unit of failure
A module in Tsak is a .tpkg: a built package with an entry point that the runtime loads into its process, giving it its own route context, its own assembly load context, and its own DI container.
The slicing criterion is simple, and it's neither "layers" nor "domains":
A module is whatever you'll want to deploy or roll back separately from everything else.
Which naturally produces a split by external protocol rather than by business entity.
┌─────────────────────────────────────────────────────────┐
│ payments.Core │
│ ──────────── │
│ • schemas: Account, Posting, BalanceSnapshot, Outbox │
│ • ledger: TransferAsync, GetBalanceAsync, Reverse │
│ • lookups: currencies, operation kinds │
│ • NOT ONE external transport │
│ │
│ Entrances: direct-vm://payments-post │
│ direct-vm://payments-balance │
└────▲──────▲──────────▲──────────▲──────────▲────────────┘
│ │ │ │ │
│ │ │ │ │ direct-vm://
│ │ │ │ │
┌────┴───┐ ┌┴──────┐ ┌─┴──────┐ ┌─┴──────┐ ┌─┴────────┐
│ .Api │ │ .Acq │ │ .Bank │ │ .Files │ │ .Recon │
│ │ │ │ │ │ │ │ │ │
│ HTTP │ │ HTTPS │ │ IBM MQ │ │ SFTP │ │ Quartz │
│ intake │ │acquirer││ bank │ │registry│ │ recon │
└────────┘ └───────┘ └────────┘ └────────┘ └──────────┘
facade outbound outbound files scheduled
What that buys in practice:
The acquirer changed its format — payments.Acq gets redeployed. The bank rail, payment intake and reconciliation don't notice: their modules never restarted.
The ledger changes least often and ships most carefully. payments.Core is the only module that touches money. Its release cadence differs from the rest, and that's fine — it also gets the fewest changes.
A new acquirer is a new module, not an edit to an existing one. It cannot break the working one because it physically lives elsewhere.
A dead SFTP server won't take down payment intake. Each module has its own route context and its own connections.
The module entry point
public static class InitRoute
{
public static IRouteContext main(IRouteContext context)
{
// Only the transports THIS module needs
context.AddComponent(new HttpComponent());
// A named store instance: payments has its own database,
// identity has its own — they share neither connections nor caches
var redb = context.GetRedbService("payments");
context.AddRouteBuilder(new PaymentsApiRouteBuilder());
return context;
}
}
The named store isn't a detail — it carries an important property: the instance is created per exchange and lives exactly as long as the message is being handled. The connection never becomes a shared resource, and concurrent requests don't queue behind a single one.
How modules call each other
Inside one worker, without the network:
public class PaymentsApiRouteBuilder : RouteBuilder
{
protected override void Configure()
{
From(Http.Listen("/api/payments/transfer").Port(8080).InOut())
.RouteId("api-transfer")
.Unmarshal(typeof(JsonMessageSerializer), typeof(TransferRequest))
.Process(RequireScope("payments:write"))
.IdempotentConsumer(e => e.Message.GetHeader<string>("Idempotency-Key"))
.To("direct-vm://payments-post") // ← into the ledger, no network
.Marshal(typeof(JsonMessageSerializer))
.Respond();
}
}
direct-vm:// is an in-process transport between route contexts. No socket, no serialization, no TLS: same thread, same exchange object.
And here's the architectural payoff: if tomorrow payments.Core has to move to its own worker, a URI string changes, not code. direct-vm://payments-post becomes rabbitmq://payments-post or grpc://.../Post and that's it. The decision of "monolith or distributed" is deferred to operations and changed by configuration.
That's arguably the whole reason to slice modules this way.
Integration routes
The outbox — and why it isn't a redb object
The one place we deliberately step away from the typed store into a flat table:
CREATE TABLE payments_outbox (
id bigserial PRIMARY KEY,
event_type text NOT NULL,
operation_id text NOT NULL,
payload jsonb NOT NULL,
created_at timestamptz NOT NULL DEFAULT now(),
processed boolean NOT NULL DEFAULT false,
processed_at timestamptz
);
CREATE INDEX ix_outbox_pending ON payments_outbox (id) WHERE processed = false;
Why, when the whole domain lives in redb: the outbox is infrastructure, not domain. It's flat by nature, lives for seconds, gets read in batches by one predicate, and gets deleted. It has no use for typing, object graphs or schema evolution — and a lot of use for a partial index on unprocessed rows and batched polling.
This is exactly the situation where it matters that the storage structure is open: you can write to redb and to plain SQL in the same transaction, because the context and the connection are shared.
await redb.Context.ExecuteAtomicAsync(async () =>
{
await redb.AddNewObjectsAsync(postings); // domain → redb
await redb.Context.ExecuteAsync( // infrastructure → SQL
"INSERT INTO payments_outbox (event_type, operation_id, payload) VALUES (@t, @o, @p)",
eventType, operationId, payloadJson);
});
The rule that falls out: redb for what you read, search and evolve. Flat SQL for what you pump through. Mixing them in one transaction is fine and normal.
Then the publisher:
From(Sql.Poll("SELECT * FROM payments_outbox WHERE processed = false ORDER BY id LIMIT 200")
.DataSource("payments")
.Delay(500)
.OnSuccess("UPDATE payments_outbox SET processed = true, processed_at = now() " +
"WHERE id = ANY(@ids)")
.Transacted())
.RouteId("outbox-publisher")
.Split(Body())
.To(Kafka.Topic("payments.events")
.Acks("All")
.EnableIdempotence(true)
.EnableTransactionalProducer(true)
.TransactionIdPrefix("payments-outbox"));
OnSuccess runs in the same transaction scope as the publish: either the event went out and is marked processed, or neither. There's no in-between state.
Outbound to an acquirer: where the save-point goes
From(Kafka.Topic("payments.events").GroupId("acq"))
.RouteId("acq-charge")
.Filter(Header("event_type").isEqualTo("transfer.completed"))
.Replayable("acq-charge") // ← SAVE-POINT
.Process(BuildAcquirerRequest)
.Retry(3).RedeliveryDelay(2000).UseExponentialBackOff()
.To(Http.Post("https://acq.example.com/v1/charge").Timeout(15_000))
.Process(ParseAcquirerResponse)
.To("direct-vm://payments-mark-charged")
.EndReplayable();
The save-point goes after entering the route and before the first external call — that is, at the boundary where state is assembled but we haven't gone outside yet. If the acquirer is down, retries don't help and the exchange fails, the snapshot lands in the dead-letter store and a support operator hits Replay from the dashboard once the acquirer is back.
An important detail the engine checks for you: don't casually put a save-point on a route where a broker or a transaction already owns redelivery. Otherwise two parties are responsible for the retry, which is the direct path to a double charge. This route deliberately isn't marked .Transacted() for exactly that reason: the save-point mechanism owns the retry here, not Kafka.
The bank on IBM MQ: here it's the opposite
From(Wmq.Queue("PAYMENTS.IN").QueueManager("QM.PROD").Transacted(true))
.RouteId("bank-inbound")
.Transacted() // ← ack and send commit together
.Unmarshal(typeof(Iso20022Codec))
.ValidateXsd(Schemas.Pacs008) // schema validation is a built-in step
.Process(MapToPostingRequest)
.To("direct-vm://payments-post")
.To(Wmq.Queue("PAYMENTS.ACK").Transacted(true));
The model here is the inverse of the previous one: the queue owns redelivery. On failure everything rolls back, the message returns to the queue, the backout counter increments, and past a threshold the message moves to a dedicated queue for triage instead of looping forever.
You write the ISO 20022 parsing (no codecs ship in the box — every profile differs), but XSD validation is built in, and that's half the work.
Registry files over SFTP
From(Sftp.Poll("/in/registry").Include("*.xml").Delay(60_000).Move("/in/done"))
.RouteId("files-registry")
.ValidateXsd(Schemas.Registry)
.Split(XPath("//Payment"))
.Threads(8) // fan the parsing out
.Process(MapToPostingRequest)
.To("direct-vm://payments-post")
.EndSplit()
.To("log://registry-done");
.Threads(8) matters here: the source polls serially, but parsing a ten-thousand-line registry fans out across a pool. Ordering within the registry is lost — acceptable for charges, not acceptable for a sequence of operations on one account. If you need that, parallelise by account, not by line.
Scheduled reconciliation
From(Quartz.Cron("0 30 3 * * ?")) // daily at 03:30
.RouteId("recon-daily")
.Process(async (e, ct) =>
{
var redb = e.Context.GetRedbService("payments", e);
var byMerchant = await redb.Query<PostingProps>()
.WhereRedb(o => o.DateCreate >= DateTime.Today.AddDays(-1))
.GroupBy(p => p.MerchantId)
.SelectAsync(g => new { g.Key, Total = Agg.Sum(g, p => p.Delta) });
e.Message.SetBody(byMerchant);
})
.To("direct-vm://recon-compare") // compare against the acquirer statement
.To(Sql.Insert("recon_report"));
The aggregation runs database-side over live data. No separate analytical stack is needed for this — you'll want one later, for something else, which we'll get to.
The route map
IN LEDGER OUT
── ────── ───
HTTP /transfer ──┐
│
IBM MQ PAY.IN ───┼──► direct-vm:// ┌──► payments_outbox
(transacted) │ payments-post ────────┤ (same transaction)
│ │ └──► postings in redb
SFTP registries ─┘ │
(Threads 8) ▼
┌─────────┐
Quartz 03:30 ────────►│ redb │
(reconciliation) │payments │
└─────────┘
▲
│
┌─────────────────────┘
│ Sql.Poll(outbox) ──► Kafka (EOS) ──┬──► acquirer (HTTP)
│ .Transacted() │ └ .Replayable
│ └──► notifications
└─ OnSuccess: processed = true
The facade: where authorization sits
The identity server runs in the same worker but on its own database. That matters: payment data and OAuth records share no connections, no transactions, no caches. Splitting them onto separate database servers later is a connection-string change, not a rewrite.
EXTERNAL CLIENTS INTERNAL MODULES
──────────────── ────────────────
browser, mobile app, payments.Api
partner payments.Acq
│ │
│ HTTPS │ direct-vm://
│ (the spec requires │ (no network,
│ a browser only for │ no TLS,
│ /authorize) │ no JSON)
▼ ▼
┌──────────────────────────────────────────────────┐
│ redb.Identity │
│ │
│ /connect/token direct-vm://identity-token │
│ /connect/introspect direct-vm://identity-... │
│ /connect/authorize ← the only one needing HTTP │
│ /scim/v2/* │
│ │
│ 79 typed audit events ──► redb + SIEM │
└──────────────────┬───────────────────────────────┘
│
┌───────▼────────┐
│ identity DB │ ← separate from payments
└────────────────┘
Authorization at the payment API entrance:
.Process(async (e, ct) =>
{
var token = e.Message.GetHeader<string>("Authorization")?["Bearer ".Length..];
// Introspection WITHOUT a network call — same process
var result = await _identity.RequestBody<IntrospectionResponse>(
IdentityEndpoints.Introspect, new { token });
if (result?.Active != true || !result.Scopes.Contains("payments:write"))
throw new UnauthorizedException();
e.SetProperty("subject", result.Subject);
})
The difference from the usual shape isn't cosmetic. Classically every internal call means a network hop to the identity server — and at meaningful traffic that becomes a bottleneck, a single point of failure, and a permanent latency tax. Here it's a method call.
From the outside it stays a normal OIDC provider: the browser parts speak HTTPS because the spec says so, and everything else — issuance, refresh, introspection, revocation, management, SCIM — is transport-neutral.
Which also implies something worth keeping in mind when designing closed segments: you can have as many front doors as you have channels. An inter-branch segment with regulator-mandated crypto, a site where only the corporate bus is allowed, a partner channel with a proprietary format — those are separate adapter modules in front of one shared core. One core, one set of permissions, one audit trail; only the adapters differ.
Cluster topology: nodes don't have to be identical
Here operations begins, and here's a property that rarely gets designed in — unfairly so.
Tsak's coordinator is three-level: cluster → group → node. A group is a geographic or logical partition. And placement is tracked per module, scoped to a group — so a node isn't a replica of its neighbour, it's a carrier of a specific module set.
cluster: production
│
├── group: acquiring ← hot path, many nodes
│ ├── node-acq-1 [payments.Api, payments.Acq, payments.Core]
│ ├── node-acq-2 [payments.Api, payments.Acq, payments.Core]
│ └── node-acq-3 [payments.Api, payments.Acq, payments.Core]
│
├── group: payouts ← bank rail, separate network
│ ├── node-pay-1 [payments.Bank, payments.Files, payments.Core]
│ └── node-pay-2 [payments.Bank, payments.Files, payments.Core]
│
└── group: reporting ← scheduled work, one node is enough
└── node-rep-1 [payments.Recon, identity]
Leader: elected cluster-wide, with epoch fencing.
A stale leader cannot corrupt state after
losing the election.
What this gives architecturally:
Contours are separated by load and criticality without being split into separate systems. The acquiring contour scales horizontally for peaks, the bank rail lives on two nodes in a separate network, scheduled work sits on one. Still one cluster, one control plane, one audit trail.
The ledger module appears in two groups. Both intake and payouts need it. That's fine — the module is stateless, state lives in the database.
The scheduler is cluster-aware. Scheduled reconciliation is marked as a leader-only job and doesn't fire on three nodes at once.
The configuration is the same on every node; node id, address and group differ:
{
"ConnectionStrings": { "Postgres": "Host=db.cluster;Database=payments;..." },
"Tsak": {
"Storage": { "Type": "Redb" },
"Cluster": {
"Enabled": true,
"ClusterName": "production",
"GroupName": "acquiring", // ← differs per group
"NodeId": "node-acq-1", // ← differs per node
"ApiEndpoint": "http://node-acq-1:9090",
"HeartbeatIntervalSeconds": 15,
"DeadNodeTimeoutSeconds": 60,
"LeaderLockTtlSeconds": 30
},
"HotReload": { "RollingUpdate": true }, // roll node by node
"Auth": { "Enabled": true }
}
}
Taking a node out for maintenance is draining, not switching off:
tsak cluster cordon node-acq-2 # takes no new work, finishes what it started
# ... maintenance ...
tsak cluster uncordon node-acq-2
Failure map: what breaks, and what handles it
An architecture is judged not by how it works but by how it fails. Walking the scenarios:
| What happened | What happens | Who fixes it |
|---|---|---|
| Process died between posting and publish | Posting is committed, outbox row isn't marked. The publisher picks it up on start | Itself |
| Duplicate message from the broker | Idempotent consumer rejects by key; if it slips through, the OperationId check under the lock catches it |
Itself, two lines |
| Acquirer returns 500 | Three retries with exponential backoff. Still failing → snapshot to the dead-letter store | Support, one click |
| Acquirer down for an hour | Same, but snapshots pile up. Once it's back — bulk replay | Support |
| Bank returned an error over MQ | Transaction rolls back, message returns to the queue, backout counter climbs; past the threshold it moves to a triage queue | Itself + triage |
| Two opposing transfers at once | Accounts locked in one canonical ascending order — no deadlock | Itself |
| Node died | Heartbeats stopped, the leader reassigned its modules to live nodes | Itself |
| The leader died | Re-election once the lock TTL expires; the epoch increments, so the old leader can't do damage | Itself |
| Balance snapshot diverged | Delete and recompute: the journal is the source of truth | Scheduled job |
| Shipped a bad module version | Upload the previous .tpkg; the old context drains, the new one starts |
One command |
| Unsigned module in the directory | Rejected at the load boundary, before a single line of its code runs | Itself |
Look at the distribution in the right-hand column: the overwhelming majority is handled by mechanisms rather than by a person. A human is needed where the failure is someone else's — and that isn't ours to fix.
Where save-points go, and where they don't
A rule worth pinning in your review checklist:
.Transacted() ─── broker/transaction owns the retry
→ do NOT add a save-point
(two parties owning retry = double charge)
.Replayable("name") ─── we own the retry
→ place it AFTER the entrance,
BEFORE the first external call
neither ─── no retry at all
→ a deliberate decision, not an oversight
The engine will warn if a save-point ends up on a transactional route — but it's better caught at review.
The decision checklist
Everything you'll have to decide, in the order to decide it. The first three are the most expensive to change later.
1. The shard key — before you need sharding. Both sides of a transfer must land on the same shard: there will be no distributed transaction. Shard by customer or account group, not by payment id.
2. Postings are append-only. No statuses, no edits. Mistakes are corrected by reversal. This also disposes of the change-history question — the journal is the history.
3. There is no balance field. Journal plus snapshot-as-cache only. The snapshot must be disposable and recomputable.
4. Idempotency on two lines. At the route, against redelivery. In the model, as the operation key inside the posting, checked under the lock.
5. One canonical lock acquisition order. Ascending id, always.
6. No network calls inside a transaction. Everything external goes behind the outbox.
7. Flat outbox, typed domain. Mixing them in one transaction is fine.
8. Slice modules by external protocol. Unit of deploy = unit of failure.
9. The ledger core has no transports. Every entrance is in-process — then extracting it into its own worker becomes a URI change.
10. The identity server gets its own database. Splitting later is a connection string.
11. Cluster groups by contour, not by sameness. Nodes carry different module sets.
12. Save-points only where we own the retry.
What's out of scope
Honest boundaries of this reference model.
Your accounting model will differ. The one shown is minimal: accounts, postings, snapshots. A real one grows a chart of accounts, analytical dimensions, multi-currency with revaluation, fund reservation and pricing rules. That's your domain, and no vendor will design it for you.
You write the financial format codecs. Transport, framing and schema validation exist; parsing a specific ISO 20022 or ISO 8583 profile is yours. It would have been yours anyway — every profile differs.
Regulated open banking needs work. Mutual-TLS client authentication, rich authorization requests, decoupled confirmation, and step-up authentication for a transaction aren't implemented. The cryptographic groundwork is in the server; this is a scoped sprint.
An analytical stack will appear eventually. Built-in aggregation covers operational queries — remaining limit, today's gap, a window for a fraud rule. A yearly slice across ten dimensions doesn't get computed on a live operational database on any storage engine, and you will build data marts. Just not in year one and not as a launch precondition — and "roll up aggregates and write them flat" is an ordinary scheduled route.
Sharding is a kit. The mechanisms exist: block key generation with the ability to move the key source into a separate database, per-connection cache isolation, several independent stores in one module. An assembled solution doesn't. That's a post of its own, in progress — including why the key authority can be an empty database with one line of DDL, and why a failure there is fixed with one SQL statement.
Wrapping up
The reference model in one paragraph: append-only postings with the balance as a function of them; a transaction containing not one network call; the outbox as the bridge into the asynchronous world; modules sliced by external protocol; a ledger core with no transports, reached in-process; a cluster whose nodes carry different module sets; and save-points exactly where we own the retry.
None of this is invention — it's discipline familiar to anyone who has built payment systems. The difference is how much code it takes to implement: transactions, locks, exact arithmetic, idempotency, an outbox, transports, save-points, clustering and an operations console are already here, and what's left is describing your ledger.
For what that layer costs if you write it yourself, see the companion post with the full breakdown.
If you're building something similar — tell me in the comments which decisions you made differently and why. The forks where you chose otherwise and didn't regret it are more useful than any feature list.
Sources and releases: github.com/redbase-app. About the store: redbase.app. Previous posts: my dev.to profile.
If this was useful — a ⭐ on GitHub helps others find it.
Top comments (0)