DEV Community: rinat kozin

redb 3.6.0: a bug report that turned out to live in six providers — plus AS2/EDI and a shared port

rinat kozin — Thu, 13 Aug 2026 14:42:31 +0000

A user's report exposed a cross-conversation leak in the core query provider — 6 of 6. Also in 3.6.0: AS2/EDI, a shared Kestrel, Camel parity.

For a year this stack grew on our own workload: we built what we needed and proved it in our own production. Since spring other people have been using it — and the mail changed character. Instead of "do you support X" we now get analyses: a reproduction, line numbers in our sources, and the workaround the reporter already wrote on their side while waiting for us.

That is the most valuable thing that can happen to a project. Thank you to everyone who wrote in rather than quietly switching to another library. One such report is the backbone of half this release — and the defect it exposed was not where it was found.

So: redb 3.6.0 — the core, the redb.Route integration engine, the redb.Tsak runtime and the redb.Identity OIDC server, all on one number, 63 packages.

The defect was two layers below where it surfaced

A developer building a chat product on redb.Route.Llm wrote: "a new person's first turn reaches the model carrying somebody else's history, and their first message gets attached into a stranger's tree."

That reads like an LLM-connector bug — conversation histories are stored as trees, something must be wrong in how a branch is loaded. We went to look, and the connector had nothing to do with it.

In the core query provider, WhereLeaves() replaced the root CTE instead of acting as a predicate on top of it. So this:

// Intent: the leaves of THIS tree
var leaves = await redb.TreeQuery<MessageProps>(root)
    .WhereLeaves()
    .ToListAsync();

actually meant "the freshest leaves of the scheme across the entire database". No root filter survived into the query at all.

The arithmetic from there is simple. In a multi-user deployment "the freshest leaf of the scheme" is the transcript of whoever wrote last, globally. Not only on the first turn — on every turn. Every conversation collapses into a single feed. Worse, the next message is attached under a parent in someone else's tree, so the data is not merely read wrong, it is written wrong.

The defect was cross-provider: six out of six — Postgres, MSSql, SQLite, each in Free and Pro. The fix is the same everywhere: tree_leaves / tree_roots are now seeded from the root instead of displacing it.

SQLite deserves its own note. The Free tier executes queries inside a native extension, so the fix lives in redb_pvt.c, not only in C#. A package carrying the previous binaries would have shipped the fix on the managed side and left the leak in Free — silently, with nothing in the log. The extension was therefore rebuilt for all three RIDs (win-x64, linux-x64, linux-arm64), and that was verified by hashing against the previous release rather than by file dates: the rebuilt Linux binaries came out byte-identical in size to the old ones, so timestamps proved nothing.

The lesson worth taking even if you never touch redb: a filter that "didn't apply" and a filter that "replaced the search space" look identical in code and differ catastrophically on data. The second one raises no error. It returns a plausible answer to a different question.

The second defect in the same report: a tool with no context

The reporter's words: "the tool receives an exchange with no request context — and the only simple way to give it a user id or tenant is to ask the model to pass it as an argument, which opens a prompt-injection escalation."

That is a precise security statement. If a tool learns on whose behalf it was invoked from an argument the model produced, then talking the model into forging that argument is the whole attack.

Before: the engine copied a hard-coded allowlist of three headers into the tool exchange and nothing else. Now:

tool routes run as a child of the agent exchange, not as a naked one — context, scope and tracing are inherited;
the run's principal and audit tags reach tools — and as the values resolved for the run, not as raw headers. The difference matters: ?user=${header.X-User-Id} and ?audit= are resolved by the engine, and a plain header copy would have missed them entirely;
.PropagateToolHeaders(...) was added — an explicit list of extra headers to forward, with a trailing * acting as a prefix match. The list is empty by default: propagation is a decision, not a convenience.

route.From("http://0.0.0.0:8080/chat")
     .Llm("llm://openai/gpt-4o?user=${header.X-User-Id}&audit=tenant:${header.X-Tenant}")
     .PropagateToolHeaders("X-Tenant", "X-Request-*")
     .To("direct://reply");

That method is why this release is a minor rather than a patch: new public API does not ship under a patch number.

AS2 and EDI, inside the route

Large retail, 3PL operators, banks running host-to-host, automotive — for twenty years they have exchanged purchase orders, invoices and shipping notices over AS2: signed and encrypted S/MIME over HTTP, with a signed receipt coming back. In .NET that has meant a commercial gateway or a Java server sitting beside your integration.

AS2 is now an ordinary redb.Route connector, schemes as2: and as2s:. Both directions, synchronous and asynchronous MDN, signed receipts, the full algorithm matrix (sha-1/256/384/512 × aes-128/192/256-cbc, 3des, optional RFC 3274 compression). Crypto is MimeKit over Bouncy Castle — the same foundation the AS2 industry interoperates on.

Interop was validated against a live OpenAS2 v4.9.0 in both directions — signed and encrypted payload, positive MDN, MIC verified.

The full walkthrough — routes, partner profiles, and how this differs from a gateway — is in the AS2 article.

One port for several connectors

An AS2 receiver is an HTTP server. You already have HTTP routes. The partner wants a single endpoint and TLS terminated once — so both have to live on the same port.

The Kestrel multiplexer (one server per host:port) used to live inside redb.Route.Http. For AS2 to use it, AS2 would have to depend on a general-purpose transport connector — dragging the whole HTTP stack into an application for the sake of one class, and inverting a sensible dependency direction.

So SharedHttpServerManager moved into its own package, redb.Route.Http.Hosting. Both connectors depend on it and not on each other; registration is idempotent and every connector resolves the same singleton. The types kept the redb.Route.Http namespace, so existing code needs no edits.

The honest part of this story is worth telling, because it generalises. The extraction landed after redb.Route.Http had already been published to nuget.org, and a published version cannot be replaced. Meanwhile redb.Route.As2 depends on Hosting but not on Http — so NuGet had no reason to ever lift Http to the fixed build, and anyone combining the old Http with As2 would quietly get two independent Kestrel managers: exactly the problem the extraction exists to prevent.

There is no clever fix for that, only an honest republish: the whole redb.Route line moved as one number, with Tsak and Identity following so their pins moved too. No combination of 3.6.0 packages can assemble the broken pair.

Camel parity: four more things

Message History — the trail an exchange leaves through a route: every node with its timing, id and label; when retries are exhausted the trail is dumped to the log, so the failing step is visible.

XSLT — .Xslt(...), .XsltContent(...) and the xslt: component. Useful exactly where it is useful: other people's formats, easier to transform with a stylesheet than with code.

Routing Slip — .RoutingSlip(...): the endpoint list is computed at runtime and the exchange is piped through it in order.

Property placeholders in endpoint URIs — {{key}} and {{key:default}}, as in Apache Camel. One route text works across three environments with no branching in code.

IBM MQ: from half a second to single-digit milliseconds

From the previous release rather than this one, but the number speaks for itself.

The default consumer polled with a blocking MQGET-WAIT on the IBM.WMQ managed client, which carries an internal ~500 ms tick that is independent of waitInterval. On an idle queue a message arrived 250–500 ms after it was actually there. The managed client exposes no public async-consume API in any 9.4.x version.

Receive was moved to an event-driven model via the XMS MessageListener, cutting latency to single-digit milliseconds. It is opt-in; polling remains the default.

Identity: e-mail uniqueness held by an index, not by a check

Registration used to work like this: look up whether a user with this address exists → if so, return 409 → if not, create. Between the check and the insert, two concurrent registrations both see "free" and both write the same address. So RequireUniqueEmail did not hold under concurrency. Login never had this problem — it is backed by a relational UNIQUE.

The guarantee is now in the database: a partial unique index UX_users_email on _users(_email) with WHERE _email IS NOT NULL. Partial is not cosmetic — null e-mails are common, and SQL Server's plain UNIQUE permits only one NULL. The address is normalised (trim + lower-invariant) before both the check and the insert, otherwise the index would treat A@x.com and a@x.com as two different people. An index violation surfaces as 409 duplicate, not as a 500.

The pre-insert check stayed — but it is now what it should always have been: a fast, deterministic early error rather than the guarantee itself.

The registration hot path also lost its debug scaffolding: a stopwatch with two dozen no-op marker calls, and a VERIFY block that fired two extra queries after every create just to feed those no-ops. Two fewer database round-trips per registration.

Tsak: a dead-letter store that did not work on PostgreSQL

The dead-letter queue is written and read through the Sql component, and on PostgreSQL it broke twice.

Capture never persisted at all: the replayable flag was bound as an int 1/0 into a BOOLEAN column, and Postgres has no implicit integer → boolean cast, so the whole INSERT threw 42804. Capture's own catch swallowed it, so dead letters were dropped on the floor.

Second: timestamps were bound as ISO-8601 strings against native timestamptz columns. There is no implicit text → timestamptz cast in a comparison operator either, so DELETE ... WHERE occurred_at < @cutoff threw 42883 — taking out the daily retention job and every date-filtered dashboard query.

Both fixes are the same idea: bind types, not text. bool → boolean/bit/0-1, DateTimeOffset → a native timestamp.

Smaller things you would notice

SumRedbAsync and AverageRedbAsync threw InvalidOperationException on an empty selection. A SUM/AVG over no rows is NULL in SQL, and the result was read through JsonElement.GetDecimal, which requires a Number. This path only became reachable after 3.5.0 made base-field aggregations honour the filter: before that WhereRedb(...) was dropped, the aggregate always spanned the whole scheme, and NULL never happened. One fix opened the road to another bug — the usual consequence of a filter that used to be silently ignored.

The fluent DSL and the URI parser disagreed on encoding: the builder encoded values differently from how the parser read them back, so values containing spaces did not survive a round trip. Both sides now use Uri.EscapeDataString.

Getting it

dotnet add package redb.Core
dotnet add package redb.Postgres        # or redb.MSSql / redb.SQLite
dotnet add package redb.Postgres.Pro    # Pro — free, no key

dotnet add package redb.Route
dotnet add package redb.Route.As2

Pro stays proprietary but free and key-less across the whole 3.x line. Everything targets .NET 9.

Sources: github.com/redbase-app. About the store itself: redbase.app.

What is next

We keep working through what comes in. The feedback journal lives in the repository, and every external signal lands there — valid, arguable and rejected alike, with our assessment and a link to wherever it was filed. If you have hit something resembling the two defects above, write in: an analysis with a reproduction is worth more than any wish list.

More of my writing: redbase.app/articles, and on dev.to.

If this was useful — a ⭐ on GitHub helps others find it.

AS2 in .NET without a separate Java gateway: EDI with trading partners, right inside the route

rinat kozin — Wed, 12 Aug 2026 18:45:36 +0000

AS2 — signed, encrypted S/MIME document exchange with trading partners — is now a native redb.Route connector, not an external gateway.

If you ship product to a big-box retailer, move freight for a 3PL, send payment advices to a bank, or exchange X12 healthcare transactions, you almost certainly move those documents over AS2. The purchase order (EDI 850), the invoice (810), the ship notice (856), the payment order (820) don't go out by email or REST — they go as a signed, encrypted S/MIME envelope over HTTP, with a signed receipt coming back. That's how regulated B2B document exchange has worked in retail, logistics, finance, manufacturing and healthcare for twenty years: Walmart, Amazon and their supplier networks, banks with a host-to-host channel, automotive, distributors — all require AS2.

In .NET, there have been two ways to do this. Either a commercial AS2 gateway — Cleo, Seeburger, BizTalk — a separate box, a separate license, a separate team to run it. Or an open-source Java server — OpenAS2, Mendelson Community — a separate JVM process next to your .NET backend, with its own inbox directory you still have to poll a document out of. Either way, AS2 lives beside your integration, not inside it.

redb.Route.As2 closes that gap: AS2 becomes an ordinary step of a route in your own .NET process. Receive an envelope from a partner, decrypt it, verify the signature, hand the document to the pipeline — validate it, transform it, drop it into Kafka or SQL — and return a signed receipt to the partner. One process, one deployment, one observability plane. Let's walk through what that looks like in code, where it's used, and why a native connector inside the ESB beats a standalone gateway.

AS2 in one minute

AS2 (Applicability Statement 2, RFC 4130) is a protocol for guaranteed delivery of business documents between two parties over the internet. Mechanically, it's an envelope:

The payload (usually EDI: X12 or EDIFACT, but it can be XML, JSON, anything) is compressed (optional), signed with your private key and encrypted with the partner's public certificate.
The finished S/MIME envelope goes to the partner as a plain HTTP POST.
The partner decrypts it with their key, verifies your signature with your certificate, and answers with an MDN (Message Disposition Notification) — a receipt. A signed MDN carries a Received-Content-MIC: a cryptographic hash of what the partner actually received.

The point of the MDN is legally meaningful non-repudiation. You sent the order, the partner returned a signed receipt whose MIC matches what you sent: you now have proof that this exact document was delivered, not some other one. That's why AS2 became the de-facto standard wherever a document carries money and obligation.

"Why not just HTTPS"

Fair question: if the channel is already TLS-protected, why sign and encrypt the document on top? Because TLS protects the channel, and AS2 protects the document. TLS lives from your socket to the partner's socket and vanishes the moment the bytes hit disk — in a proxy log, in an inbox directory, on a load balancer the document is already in the clear. The AS2 S/MIME envelope stays signed and encrypted the whole way and at rest — only the private-key holder can decrypt it, and the signature proves the author. And crucially, TLS gives you no receipt: HTTPS has no MDN with a MIC, and that's what makes delivery non-repudiable. AS2 usually runs over HTTPS (as2s) anyway — the two don't compete: TLS encrypts the channel, S/MIME provides the document and the non-repudiation.

A readable route: the endpoint is a string

redb.Route is Apache Camel for .NET: a route is described as From → … → To, and an endpoint is a URI string. The AS2 connector adds two schemes, as2 (HTTP) and as2s (HTTPS), and they read like a sentence:

as2s://partner.example.com/as2?connectionFactory=walmart          # who we send to
as2:/inbound/orders?host=0.0.0.0&port=4080&connectionFactory=walmart   # where we receive
as2:/as2/mdn?host=0.0.0.0&port=4081&mode=mdn&connectionFactory=walmart # where the partner posts an async MDN

The URI states the intent up front: where we're going, what port we listen on, which partner. Certificates and algorithms don't live in the string — they don't belong there. There's a dedicated object for that.

A partner is one object, not a scatter of parameters

An AS2 exchange is always an agreement between two sides: whose certificates, which AS2 identifiers, what to sign and encrypt with, which MDN mode. All of that is an As2ConnectionFactory, registered once by name:

context.AddToRegistry("walmart", new As2ConnectionFactory
{
    OurCertificate     = ourPfx,     // our cert + PRIVATE key — signs outgoing, decrypts incoming
    PartnerCertificate = theirCer,   // partner's PUBLIC cert — encrypts outgoing, verifies their signature
    As2From = "OUR-AS2-ID",
    As2To   = "WALMART-AS2-ID",
    PartnerUrl = "https://partner.example.com/as2",

    // Profile — what both sides agreed on
    Sign = true, Encrypt = true, Compress = false,
    SignAlg = "sha-256", EncryptAlg = "aes-128-cbc",
    SignedMdn = true, MdnMode = As2MdnMode.Sync,
});

Routes reference the partner by name — ?connectionFactory=walmart in the URI, or .ConnectionFactory("walmart") in the fluent DSL. Add a second partner and you register another object; the route URI doesn't change. Certificates are versioned with your application, not sitting in a gateway's keystore that one person on the team knows about.

One route, three environments

The endpoint URI is a string, and {{key}} placeholders work inside it — redb.Route resolves them from IConfiguration when the route is built. The partner's address differs across dev, staging and prod, but the route stays the same:

From("direct://outbound")
    .To(As2.Send("{{walmart.as2.url}}").ConnectionFactory("walmart"));

# appsettings.Development.json → "walmart.as2.url": "https://sandbox.partner/as2"
# appsettings.Production.json  → "walmart.as2.url": "https://edi.partner.com/as2"

Host, receiver port, partner name — all externalized to configuration instead of hard-coded into the route. Secrets — the PFX password — come from the same layer (environment variables, user-secrets) and never settle into source. One built image ships from dev to prod; only the config changes.

Sending (producer)

From("direct://outbound")
    .To(As2.Send("https://partner.example.com/as2").ConnectionFactory("walmart"));

Behind that single line: compression (if enabled), signing with your key, encryption with the partner's certificate, the POST, and parsing the MDN that comes back. The connector puts the outcome on exchange.Out so the route can act on it:

Header on `exchange.Out`	Meaning
`redbAs2.mdnDisposition`	the partner's verdict, e.g. `automatic-action/MDN-sent-automatically; processed`
`redbAs2.signatureValid`	`bool` — the MDN's signature verified
`redbAs2.mdnMicMatch`	`bool` — the partner received exactly what we sent, intact

And right there, in the same route, you branch on the result:

From("direct://outbound")
    .To(As2.Send("https://partner/as2").ConnectionFactory("walmart"))
    .Choice()
        .When(e => e.Out!.GetHeader<bool>(As2Headers.MdnMicMatch))
            .Log("delivered & verified")
        .Otherwise()
            .To("direct://delivery-alert");   // MIC mismatch or negative MDN — escalate

The connector does not throw on a negative MDN or a MIC mismatch — it surfaces the fact and lets the route decide. For one partner a MIC mismatch is cause to raise an alert immediately; for another, log it and move on. That's your process's policy, not behavior wired into a gateway.

Receiving (consumer / AS2 server)

From(As2.Receive("/inbound/orders").Host("0.0.0.0").Port(4080).ConnectionFactory("walmart"))
    .Unmarshal(...)                 // your EDI parsing
    .To("direct://process-order");

From(As2.Receive(...)) stands up an AS2 server. The incoming envelope is decrypted with your private key, its signature verified with the partner's certificate, unpacked — and the route receives the clean business document. Its real content type is on Message.ContentType (e.g. application/edi-x12), and the exchange metadata sits under redbAs2.*:

Header	Meaning
`redbAs2.mic` / `redbAs2.micalg`	the computed Message Integrity Check and its algorithm
`redbAs2.signatureValid`	the inbound signature verified against the partner cert
`redbAs2.remoteAddress`	the sender's IP
`redbAs2.partner`	the resolved connection-factory name

The connector builds and returns a synchronous MDN itself. The AS2 wire headers (AS2-From, AS2-To, Message-ID, Subject) are copied onto the message as-is. The S/MIME wrapper Content-Type is deliberately not leaked into the headers — the route sees the type of the real document, not the transport envelope.

Asynchronous MDN

Large partners often require an asynchronous receipt: accept the message, answer 200, and post the signed MDN separately later. The sender registers the outgoing Message-ID, and the inbound MDN is correlated by Original-Message-ID:

// In the partner profile
MdnMode = As2MdnMode.Async,
AsyncMdnUrl = "https://our-host:4081/as2/mdn",

// Routes
From(As2.Receive("/inbound").Host("0.0.0.0").Port(4080).ConnectionFactory("walmart"))
    .To("direct://process");

From(As2.ReceiveMdn("/as2/mdn").Host("0.0.0.0").Port(4081).ConnectionFactory("walmart"))
    .Process(e =>
    {
        var original = e.In.GetHeader<string>(As2Headers.MessageId);   // which of our documents is acknowledged
        var ok = e.In.GetHeader<bool>(As2Headers.MdnMicMatch);         // and acknowledged intact
    });

A dedicated endpoint for inbound MDNs is just another From in a route, and the arriving receipt becomes a message you can act on: update the order status, clear a retry, close a saga.

Algorithm matrix

Partners agree on specific algorithms, and the connector supports the standard set:

Knob	Values
`SignAlg`	`sha-1`, `sha-256` (default), `sha-384`, `sha-512`
`EncryptAlg`	`aes-128-cbc` (default), `aes-192-cbc`, `aes-256-cbc`, `3des`
`Compress`	`true` / `false` (RFC 3274)

An unsupported algorithm fails fast when the route is built, not at run time on the first message to a partner. A configuration error is visible at startup, not at 3 a.m. in the dead-letter logs.

The crypto underneath is MimeKit (on Bouncy Castle) — the same cryptographic foundation the entire AS2 world interoperates on: Apache camel-as2, OpenAS2 and Mendelson all sit on Bouncy Castle. We land on the shared foundation the parties actually interoperate over, rather than inventing our own S/MIME.

How it differs from a gateway

.NET projects have covered AS2 three ways. The difference isn't "can or can't" — everyone can, it's one protocol. The difference is where AS2 lives relative to your logic.

	Commercial gateway	Java server (OpenAS2/Mendelson)	redb.Route.As2
Process	separate box	separate JVM next door	your .NET process
Received document	file in an inbox dir	file in an inbox dir	a message in the route
How to pick it up	a poller job	a poller job	straight into the pipeline
Deployment	its own install	its own install	with your app
Observability	its own panel	JVM logs	shared traces and stats
Further processing	outside the gateway, by hand	outside the server, by hand	the same EIPs in the same route
License	paid	open	open, no key

A standalone gateway is justified when the AS2 boundary is deliberately isolated — in a DMZ run by a different team, say. But when the document goes to your .NET backend for processing anyway, the intermediate box is an extra hop, an extra directory, and an extra component in the audit diagram.

Where it's used, and why a native connector

AS2 is needed wherever a document carries an obligation and the partner dictates the channel. A few typical situations.

Supplier to a big-box retailer. To ship to Walmart, Target or Amazon Vendor, a supplier must accept orders (850) and send invoices (810) and ASNs (856) over AS2 with a signed MDN. This used to mean a standalone gateway; now From(As2.Receive(...)) accepts the order straight into the route that validates it and lands it in your ERP.

3PL and logistics. A warehouse and a carrier exchange shipment statuses, receipt confirmations, inventory. The volume is streaming, and keeping a separate Java box with an inbox directory that a cron job scoops documents out of is an extra link. The document should flow, not sit in a folder.

Healthcare. X12 HIPAA transactions (claims, remittance) between payers and providers go over AS2 with strict signing and encryption requirements. Same profile — sign, encrypt, signed MDN.

Finance and payments. EDI payment and settlement documents — the payment order and remittance advice (X12 820/824) — travel between corporate clients and their banks over AS2 where the bank offers a host-to-host channel. AS2 doesn't replace SWIFT or EBICS on the interbank rails here; it covers the EDI layer: corporate-to-bank document flow and payment advices in the supply chain, where payment is just another EDI document alongside the order and the invoice. What matters to fintech is exactly AS2's non-repudiation: a signed MDN with a MIC is proof the bank received this exact payment order, not another.

Manufacturing and supply chain. Automotive and industrial supply-chain networks push orders, delivery schedules and ship notices (EDIFACT/X12) between OEMs and their tiered suppliers. The exchange is mandatory: without electronic document exchange, a supplier simply isn't onboarded.

Insurance. Enrollment, claims, remittance (X12) between payers, brokers and providers — the same signing-and-encryption profile, the same receipt requirements.

Consolidating a gateway. You already have a commercial AS2 gateway, but it's a separate box with its own license, its own patch cycle and its own monitoring, running alongside the .NET backend that does all the business logic. The AS2 connector folds that boundary into the application.

Why a native connector in the ESB rather than a gateway beside it:

The document flows, it doesn't sit. With a standalone gateway, a received file lands in an inbox directory you then have to pick up. With the connector, a received document is a message in a route: validation, transformation, routing right away. No intermediate folder and no scoop-up job.
One process, one deployment. No second box to install, patch, monitor and explain to an auditor. The AS2 endpoint stands up on the process's shared Kestrel host along with the rest of its HTTP routes.
One observability plane. AS2 endpoints emit statistics and health (visible in the redb.Tsak dashboard) and distributed traces alongside every other connector: the producer opens a Client span, the consumer a Consumer span linked over the inbound W3C traceparent. The exchange with a partner shows up in the same Jaeger trace as the document's onward path.
Composition with EIPs. This is the main thing. AS2 isn't an island, it's a step. A received document can go through the full redb.Route pattern catalog: validate against a schema, transform with XSLT, split, enrich, tee off to an archive with WireTap, land in Kafka and SQL at once.

AS2 as one step of an end-to-end route

The value of the connector shows when AS2 stands not by itself but in a chain. Accept an order from a partner, transform it, split it and route it onward — in one route:

From(As2.Receive("/inbound/orders").Host("0.0.0.0").Port(4080).ConnectionFactory("walmart"))
    .Validate(...)                               // document matches the schema
    .Xslt("styles/x12-to-canonical.xsl")         // X12 → your canonical model
    .WireTap("sftp://archive/edi?...")           // a copy to the archive with retention
    .To("kafka://orders?brokers=...")            // onto the bus for processing
    .To("sql:INSERT INTO inbound_orders ...");   // and into the audit DB

The same document that arrived in an encrypted AS2 envelope ends up — in one pass — validated, transformed, archived, in Kafka and in SQL, and all of it visible in a single trace. A standalone gateway would have handed you a file in a folder; here you get a full pipeline whose entry point happens to be AS2.

The outbound direction is symmetric: assemble a document from your system, send it to the partner, parse the MDN, update the status — one route again.

The full loop: order in, invoice out

Onboarding a supplier to a retailer is two AS2 directions, living as two routes in one process.

Inbound — the order (EDI 850) from the retailer:

From(As2.Receive("/inbound").Host("0.0.0.0").Port(4080).ConnectionFactory("walmart"))
    .Validate(typeof(X12OrderValidator))          // 850 structure is valid
    .Unmarshal(typeof(X12Format), typeof(Order))  // X12 → domain model
    .Process(async (e, ct) => await _orders.Accept((Order)e.In.Body!, ct))
    .To("kafka://orders-inbound?brokers={{kafka.brokers}}");

The order is accepted, validated, parsed, stored and published to the bus in one pass, and a synchronous MDN went back to the retailer automatically. Your system then processes the order at its own pace.

Outbound — the invoice (EDI 810) back to the partner once the shipment is ready:

From("kafka://invoices-ready?brokers={{kafka.brokers}}")
    .Marshal(typeof(X12Format))                   // domain model → X12 810
    .To(As2.Send("{{walmart.as2.url}}").ConnectionFactory("walmart"))
    .Choice()
        .When(e => e.Out!.GetHeader<bool>(As2Headers.MdnMicMatch))
            .Process(e => _invoices.MarkDelivered(e))
        .Otherwise()
            .To("direct://edi-ops-alert");

Both directions use one connectionFactory("walmart") — the same certificates, the same profile. The whole EDI relationship with a partner is two routes and one partner object, inside your application, under your observability.

Visible in one trace

AS2 in a standalone gateway is a black box: the partner says "it didn't arrive," and you go digging in someone else's logs on someone else's box. Here, AS2 endpoints are part of redb.Route's shared observability. The producer opens a Client span, the consumer a Consumer span linked to the inbound document over the W3C traceparent. In one Jaeger trace you see the whole thing: envelope arrived → decrypted and signature verified → validated → transformed → sent to Kafka. Endpoint statistics — how many received, how many sent, errors — sit in the redb.Tsak dashboard next to every other connector. When a partner disputes delivery, you have the trace for a specific Message-ID, not "somewhere in the gateway logs."

What to know about AS2 in practice

A few things you step on exactly once.

The MIC is computed over the canonical form, byte for byte. The Message Integrity Check is a hash of the signed part in CRLF-canonical form, computed exactly the way the sending side computed it. That's where all of AS2's interop pain lives: if canonicalization or the algorithm diverges by even a byte, the MIC in the MDN won't match and the partner decides it got the wrong thing. The connector computes the MIC per RFC 4130 and checks it on receive and in MDN parsing — and that's exactly what's verified against an external server, not against itself.

Sync or async is the partner's call, not yours. A synchronous MDN is simpler: the receipt is in the same response, the route knows the outcome immediately. But at volume a synchronous MDN holds the HTTP connection until processing finishes, and large partners require async: accept, answer 200, post the MDN separately. The connector supports both; the choice is a line in the partner profile, not a route rewrite.

A signed MDN is about non-repudiation, not a checkbox. SignedMdn = true means the receipt is signed with the receiving side's key and carries a MIC. For a document that carries an obligation, that's the proof: you hold a signed confirmation that the partner received this exact document, intact. An unsigned MDN is just "it arrived," which is worth little in a dispute.

Content-Transfer-Encoding: binary. EDI documents are binary in spirit, and partners typically exchange in binary rather than base64 — less overhead at volume. A profile detail, but exactly the one home-grown implementations trip on.

Certificates and secrets

AS2 rests on a key pair: your private key (signs outgoing, decrypts incoming) and the partner's public certificate (encrypts outgoing, verifies their signature). In the connector that's an X509Certificate2 on the As2ConnectionFactory — load it from a PKCS#12/PFX however suits you: a file, the Windows certificate store, a secrets manager.

Certificates are part of the partner's configuration in the registry, not strings in a URI. The PFX password, if it comes through an endpoint parameter, is marked [Sensitive] and redacted from logs and the dashboard — the secret doesn't leak into a trace. Rotating a partner certificate is swapping the object in the registry; the route isn't touched.

Interop proven, not claimed

It's easy to write "AS2 supported" and attach unit tests that run the connector against itself. That proves internal consistency, but not that your envelope is accepted by software you didn't write. The real test is exchanging with an independent implementation.

The connector is verified against a live OpenAS2 v4.9.0 (a mature, Bouncy-Castle-based server) in Docker, in both directions, with the profile "SHA-256 sign + AES-128-CBC encrypt + signed MDN":

redb → OpenAS2. Our producer sent a document; OpenAS2's own logs confirmed it decrypted our envelope, verified our signature, stored the document and returned a positive signed MDN. Our MDN parser then verified that MDN's signature and confirmed the Received-Content-MIC matched what we sent.
OpenAS2 → redb. OpenAS2 built a signed, encrypted document and sent it to our consumer. We decrypted it with our private key, verified OpenAS2's signature, handed the EDI to the route, and returned a signed MDN that OpenAS2 accepted and correlated with its pending message.

That closes AS2's "hard part" in both directions: MIC computation per RFC 4130, S/MIME structure and MDN handling — send and receive — are correct against a real, independent partner, not merely self-consistent.

The tests are open. How exactly it was verified — three layers (crypto round-trips and MIC, an end-to-end producer→consumer loop over a live Kestrel, interop against OpenAS2) — is laid out in the connector's TESTING.md on GitHub. The interop harness (a docker-compose with OpenAS2, its config and generated certificates) reproduces with one command; the interop test is gated, so an ordinary run is green without the container.

FAQ

Multiple partners on one port? Yes. Different partners' receive endpoints live on the shared Kestrel host, separated by path (/inbound/walmart, /inbound/target), each with its own connectionFactory. One server, several partnerships.

Large files? The message body is bytes, and it flows through the normal HTTP pipeline rather than being assembled into a string. For genuinely heavy transfers partners usually move to SFTP — which in redb.Route is also a connector, and sits in the same route.

mTLS / client certificates on the transport? AS2 itself authenticates the document by signature, but some partners additionally require TLS client-auth. The as2s scheme runs on the same Kestrel host as the HTTP connector, with its TLS settings.

What about restarts? An async MDN is correlated by Original-Message-ID; outgoing messages awaiting a receipt are your route's state, held where you keep saga or idempotency state, not in the connector's memory.

Install

services.AddRedbRoute(route =>
{
    route.Services.AddRedbRouteAs2();
    route.AddRouteBuilder<MyRoutes>();
});

AddRedbRouteAs2() registers the as2 / as2s schemes and reuses the process's shared Kestrel host — an AS2 route and a plain HTTP route in the same worker share one server and never fight over a port.

The package is redb.Route.As2 on NuGet; the source and the full DSL reference are in the connector README. AS2 is one more transport in the redb.Route family, alongside Kafka, RabbitMQ, IBM MQ, SFTP and the rest: the same From → … → To, the same EIPs, the same observability. The only difference is that the input and output are a signed, encrypted envelope your trading partner is waiting for.

More of my writing: redbase.app/articles, and on dev.to.

If this was useful — a ⭐ on GitHub helps others find it.

A real local database in Blazor WebAssembly and MAUI: no EF DbContext, no Include, no migrations

rinat kozin — Sun, 09 Aug 2026 14:33:56 +0000

Offline storage for complex graphs and an outbox queue: quick start for MAUI and the browser, everyday operations, and why EF Core just gets in the way here.

A delivery driver walks into a basement loading dock and the signal dies. A warehouse clerk spends
three hours in the one corner where Wi-Fi comes and goes. A user fills in a four-screen form and hits
Save at the exact moment your API goes down for a deploy.

The app has to keep working. Which means it needs its own database on the device — not a response
cache, but real local storage holding half-finished documents, local state, and a queue of changes
waiting to go out. The outbox that flushes itself once the connection comes back.

And that is where the fun starts.

Sound familiar?

"We'll just use key-value, it's simple." localStorage, Preferences, a JSON file. Works
beautifully right up until someone asks for "unfinished orders from the last week, sorted by
priority." Answering that means loading everything into memory and looping. Fine at two hundred
records. At ten thousand, the phone gets warm.

"Let's use EF Core, everyone knows it." And now you have migrations on the client. The app
updates, and a migration has to run on a user's device, against their data, with nobody watching. If
it goes sideways, you find out from an app store review. Worse, it is not a one-time tax: local state
churns far more than server entities, because it is drafts, wizard steps, sync flags.

"The graph is complex anyway." An order with line items, each with attachments and a status
history, plus a customer with an address. On the server you normalized that and wrote Include /
ThenInclude. On the client you need the whole thing every time — the user opened a draft, show me
everything. Miss an Include and you get null where data should be, or an N+1 out of nowhere.

"Fine, serialize the graph into a JSON column." The classic escape hatch: complex stuff goes to
text, simple stuff stays in columns. The graph persists, but querying is over — "status = draft and
total > 10,000" is now a full scan in memory. And your type safety is reduced to hoping
JsonSerializer never meets a field it does not recognize.

Now multiply that by the size of a real app. It is not one entity, it is hundreds of classes, and
each one has local state of its own: form drafts, wizard position, active filters, a snapshot for
undo, a cached response under a specific key. And that state is not flat — it has nested objects,
collections, statuses.

A table per state means hundreds of tables and hundreds of migrations shipping to users' phones. One
"key → JSON" table instead is fast to write and comfortably familiar, except:

type safety is gone: rename a property, and the old JSON quietly deserializes with a null, and the bug shows up a week later on someone else's device;
search is impossible: "show me unfinished drafts above the limit" means pulling everything out and sifting through it in memory;
and reading that pile with your own eyes is not a great time either.

Familiar bottom line: half the local-storage code is not business logic, it is bookkeeping for how you
store things.

What it looks like instead

The schema is a plain C# class. No DbContext, no migration files, no Include.

[RedbScheme("Order")]
public class OrderProps
{
    public string Number { get; set; } = "";
    public OrderStatus Status { get; set; }
    public decimal Total { get; set; }
    public DateTime CreatedAt { get; set; }

    public Customer? Customer { get; set; }               // nested object
    public List<OrderItem> Items { get; set; } = new();   // nested collection
    public string[]? Tags { get; set; }
}

Save the whole graph in one line. Load the whole graph in one line. Query nested fields with LINQ that
runs in the database, not in memory:

await redb.SaveAsync(order);                       // the whole graph, Items and Customer included

var draft = await redb.LoadAsync<OrderProps>(id);  // the whole graph back, no Include

var pending = await redb.Query<OrderProps>()
    .Where(o => o.Status == OrderStatus.Draft && o.Total > 10000m)
    .OrderByDescending(o => o.CreatedAt)
    .ToListAsync();

Add a property to the class and it simply shows up. Nothing to migrate: no migration files, no
ALTER TABLE, and objects saved earlier keep loading fine.

And this is not a JSON blob: every property lives in a typed, indexed column, so the condition
above is a real SQL filter rather than a scan. Strong typing survives all the way down — nested
objects, collections, dictionaries.

As for those hundreds of classes: you never list them anywhere. Tag them with [RedbScheme] and
startup finds them in the assembly and sets up the schemas:

// one line for the entire app — for the first class and for the three-hundredth
await redb.InitializeAsync(ensureCreated: true, typeof(OrderProps).Assembly);

A new kind of state is a new class in your code and nothing else. No table, no migration, no registry
entry.

Underneath it is ordinary SQLite — the same file you were going to ship anyway. And the same code runs
on a server over PostgreSQL or SQL Server, which means the client and the backend end up sharing one
model.

What follows is a quick start for both, the everyday operations, and an honest comparison. Not a word
about how the provider works inside — this is a piece about using it.

The model for the examples

To keep the code short, everything below uses a note. It all works the same on the graph from the
first example, it just reads better this way:

[RedbScheme("Note")]
public class NoteProps
{
    public string Title { get; set; } = "";
    public string Body { get; set; } = "";
    public int Priority { get; set; }
    public DateTime CreatedAt { get; set; }
    public string[]? Tags { get; set; }
}

RedBase ships three providers: PostgreSQL, SQL Server and SQLite. On the client it is SQLite.

Which package to install

The SQLite provider comes in two editions, and on the client there is effectively no choice to make.

	`redb.SQLite`	`redb.SQLite.Pro`
Implementation	part of the logic in a native SQLite extension	pure C#
Server, desktop	yes	yes
Blazor WebAssembly	no	yes
Android, iOS	no	yes

The Free edition keeps part of its logic in a native SQLite extension, and browsers cannot load those;
on mobile that extension is not built at all. Pro is C# from top to bottom, so it runs everywhere.

Pro is free and needs no license key — the whole 3.x line, commercial production included. The
package is closed-source, but there is nothing to pay for and nothing to activate: install it and go.

dotnet add package redb.SQLite.Pro

Nothing else to add — redb.Core, SQLite itself and the rest come in transitively. You need .NET 8, 9
or 10. Everything below was verified on 3.5.0.

Quick start: mobile (MAUI)

Mobile first, because it is the easy one: the database is a regular file that survives restarts on its
own.

Step 1. Project and package

dotnet workload install maui-android
dotnet new maui -n MyApp
cd MyApp
dotnet add package redb.SQLite.Pro

Building Android-only from Windows? Drop the ios and maccatalyst entries from <TargetFrameworks>,
or restore will demand a workload you do not have.

Step 2. Registration in `MauiProgram.cs`

using redb.Core.Models.Configuration;
using redb.Core.Pro.Extensions;      // AddRedbPro
using redb.SQLite.Pro.Extensions;    // UseSqlite

public static MauiApp CreateMauiApp()
{
    var builder = MauiApp.CreateBuilder();
    builder.UseMauiApp<App>();

    // AppDataDirectory is the app's private folder: the file survives restarts
    // and updates, and goes away when the app is uninstalled.
    var dbPath = Path.Combine(FileSystem.AppDataDirectory, "app.db");

    builder.Services.AddRedbPro(options => options
        .UseSqlite($"Data Source={dbPath}")
        .Configure(c => c.PropsSaveStrategy = PropsSaveStrategy.ChangeTracking));

    builder.Services.AddSingleton<RedbBootstrap>();
    builder.Services.AddSingleton<MainPage>();

    return builder.Build();
}

PropsSaveStrategy.ChangeTracking means "only write the properties that actually changed" instead of
rewriting the whole object. On a phone that saves both time and flash wear.

Step 3. Initialize exactly once

Here is the first thing that trips people up. On a server this happens by itself when the host starts.
MAUI does not run hosted services, so you have to call it yourself:

var redb = services.GetRequiredService<IRedbService>();

// Creates the database structure if it is missing, and registers schemas for every
// [RedbScheme] class in the given assembly. You do not enumerate them.
await redb.InitializeAsync(ensureCreated: true, typeof(NoteProps).Assembly);

You can omit the assembly and let it scan everything loaded. On a client, name it explicitly — faster
and more predictable.

The second trap: Android recreates the Activity on rotation and when you come back from the
background. Tie initialization to a page event and it runs several times. Tie it to the process
instead — Lazy<Task> does that in one line and behaves under concurrent callers:

public sealed class RedbBootstrap
{
    private readonly Lazy<Task> _init;

    public RedbBootstrap(IServiceProvider services)
    {
        _init = new Lazy<Task>(async () =>
        {
            var redb = services.GetRequiredService<IRedbService>();
            await redb.InitializeAsync(ensureCreated: true, typeof(NoteProps).Assembly);
        });
    }

    /// Call before touching the database. Actually runs once per process.
    public Task EnsureInitializedAsync() => _init.Value;
}

Step 4. The page

public partial class MainPage : ContentPage
{
    private readonly IRedbService _redb;
    private readonly RedbBootstrap _bootstrap;

    public MainPage(IRedbService redb, RedbBootstrap bootstrap)
    {
        InitializeComponent();
        _redb = redb;
        _bootstrap = bootstrap;
    }

    protected override async void OnAppearing()
    {
        base.OnAppearing();
        await _bootstrap.EnsureInitializedAsync();

        CountLabel.Text = $"Notes: {await _redb.Query<NoteProps>().CountAsync()}";
    }
}

That is the whole setup. Run dotnet build -f net10.0-android -t:Run, the app opens, the database is
created on first launch and stays on the device until the app is uninstalled.

Release builds use trimming and AOT — the provider handles that, nothing extra to configure. If you
crank trimming past the defaults, root the assembly holding your schema classes: they are read through
reflection and the linker does not know about them.

Quick start: Blazor WebAssembly

The same code runs in a browser, with three wrinkles. All three fail the same unhelpful way — the
project builds clean and then breaks in the browser — so it is worth knowing all of them.

Wrinkle 1. The build needs an extra tool

dotnet workload install wasm-tools

<PropertyGroup>
  <WasmBuildNative>true</WasmBuildNative>
</PropertyGroup>

Why: browsers have no OS library loader, so SQLite has to be compiled into the runtime at build
time rather than shipped next to it. Release turns this on by itself; dotnet run and Debug need the
flag. Without it you get a stock runtime with no SQLite in it, and a crash on the first database call.

The first build after this gets noticeably slower — that is the native link step, and it is a one-off.
Incremental builds stay quick.

Wrinkle 2. Initialization is manual here too

Same as MAUI, same reason: WebAssemblyHost does not run hosted services.

Wrinkle 3. Persistence is on you

The browser's file system in .NET is memory. While the tab is open the database behaves normally; hit
reload and it is gone. RedBase does not hand you a persistence mechanism — deliberately, because the
right answer depends on the app: IndexedDB, the Cache API, or OPFS.

Here is a working IndexedDB version. It needs no special build flags and uses the ordinary File API.

One catch that makes naive implementations look fine while they lose data. SQLite in the browser
runs in WAL mode: recent commits land in the app.db-wal companion while the main file stays nearly
empty. Save just app.db and you get a database that "restores" and turns out empty. Both files have
to travel.

wwwroot/js/dbPersistence.js:

const DB_NAME = "myapp-db";
const STORE = "files";

function openIdb() {
    return new Promise((resolve, reject) => {
        const req = indexedDB.open(DB_NAME, 1);
        req.onupgradeneeded = () => req.result.createObjectStore(STORE);
        req.onsuccess = () => resolve(req.result);
        req.onerror = () => reject(req.error);
    });
}

export async function load(key) {
    const db = await openIdb();
    try {
        return await new Promise((resolve, reject) => {
            const tx = db.transaction(STORE, "readonly");
            const req = tx.objectStore(STORE).get(key);
            req.onsuccess = () => resolve(req.result ? new Uint8Array(req.result) : null);
            req.onerror = () => reject(req.error);
        });
    } finally { db.close(); }
}

export async function save(key, bytes) {
    const db = await openIdb();
    try {
        await new Promise((resolve, reject) => {
            const tx = db.transaction(STORE, "readwrite");
            tx.objectStore(STORE).put(new Uint8Array(bytes), key);
            tx.oncomplete = () => resolve();
            tx.onerror = () => reject(tx.error);
            tx.onabort = () => reject(tx.error);
        });
    } finally { db.close(); }
}

Services/SqliteFilePersistence.cs:

using Microsoft.JSInterop;
using redb.Core.Data;

public sealed class SqliteFilePersistence
{
    private readonly IJSRuntime _js;
    private readonly string _dbPath;
    private IJSObjectReference? _module;

    public SqliteFilePersistence(IJSRuntime js, string dbPath)
    {
        _js = js;
        _dbPath = dbPath;
    }

    private async Task<IJSObjectReference> ModuleAsync()
        => _module ??= await _js.InvokeAsync<IJSObjectReference>("import", "./js/dbPersistence.js");

    /// Pull the database out of IndexedDB. Must happen before anything touches the
    /// database, or SQLite creates an empty file and there is nothing left to restore.
    public async Task RestoreAsync()
    {
        var module = await ModuleAsync();

        foreach (var path in new[] { _dbPath, _dbPath + "-wal" })
        {
            var bytes = await module.InvokeAsync<byte[]?>("load", path);
            if (bytes is { Length: > 0 })
                await File.WriteAllBytesAsync(path, bytes);
        }
    }

    /// Push the current state back into IndexedDB.
    public async Task PersistAsync(IRedbContext context)
    {
        // PASSIVE matters. TRUNCATE wants an exclusive lock, and on a single-threaded
        // browser there is nobody to release it — the call just hangs.
        try { await context.ExecuteAsync("PRAGMA wal_checkpoint(PASSIVE);"); } catch { }

        var module = await ModuleAsync();

        foreach (var path in new[] { _dbPath, _dbPath + "-wal" })
        {
            if (File.Exists(path))
                await module.InvokeVoidAsync("save", path, await File.ReadAllBytesAsync(path));
        }
    }
}

Program.cs — order matters here:

const string DbPath = "/app.db";

var builder = WebAssemblyHostBuilder.CreateDefault(args);
builder.RootComponents.Add<App>("#app");
builder.RootComponents.Add<HeadOutlet>("head::after");

builder.Services.AddSingleton(sp =>
    new SqliteFilePersistence(sp.GetRequiredService<IJSRuntime>(), DbPath));

builder.Services.AddRedbPro(options => options.UseSqlite($"Data Source={DbPath}"));

var host = builder.Build();

// 1. Restore the files first...
await host.Services.GetRequiredService<SqliteFilePersistence>().RestoreAsync();

// 2. ...and only then touch the database.
var redb = host.Services.GetRequiredService<IRedbService>();
await redb.InitializeAsync(ensureCreated: true, typeof(NoteProps).Assembly);

await host.RunAsync();

Persisting ships the whole file, so do not call it on every write. Sensible moments: after a
meaningful user action, on a timer, on beforeunload.

await Redb.SaveAsync(note);
await Persistence.PersistAsync(Context);   // debounce this in a real app

Everyday operations

From here on it is identical on mobile and in the browser. Get the service from DI:

@inject IRedbService Redb

Create

An object is a RedbObject<T> wrapper plus your data in Props. The wrapper has service fields; the
only one you need on day one is name, a human-readable label.

var note = new RedbObject<NoteProps>
{
    name = "Buy milk",
    Props = new NoteProps
    {
        Title = "Buy milk",
        Body = "And bread",
        Priority = 2,
        CreatedAt = DateTime.UtcNow,
        Tags = ["home", "groceries"]
    }
};

long id = await Redb.SaveAsync(note);

SaveAsync hands back the id, and also stamps it onto the object, so note.Id is populated after the
call.

Got several objects? Do not loop. The same method takes a collection and writes it in one batch,
returning the ids:

var notes = new List<RedbObject<NoteProps>> { note1, note2, note3 };

List<long> ids = await Redb.SaveAsync(notes);

Read by id

var loaded = await Redb.LoadAsync<NoteProps>(id);

if (loaded is not null)
{
    Console.WriteLine(loaded.Props.Title);      // "Buy milk"
    Console.WriteLine(loaded.Props.Tags![0]);   // "home"
}

The object comes back whole, arrays and nested objects included. There is no "forgot to load the
related data" failure mode here — properties are always there.

If no object has that id you get null, hence the check.

Update

There is no separate Update. Change the loaded object and save it again:

var note = await Redb.LoadAsync<NoteProps>(id);

note.Props.Priority = 5;
note.Props.Body = "And bread, and yogurt";

await Redb.SaveAsync(note);

With PropsSaveStrategy.ChangeTracking, only those two properties hit the database.

Delete

await Redb.DeleteAsync(note);

Queries

Plain LINQ. Conditions run in the database, not in memory:

// high-priority notes, newest first
var important = await Redb.Query<NoteProps>()
    .Where(n => n.Priority >= 3)
    .OrderByDescending(n => n.CreatedAt)
    .ToListAsync();

// substring search
var found = await Redb.Query<NoteProps>()
    .Where(n => n.Title.Contains("milk"))
    .ToListAsync();

// date range plus a compound condition
var lastWeek = DateTime.UtcNow.AddDays(-7);
var recent = await Redb.Query<NoteProps>()
    .Where(n => n.CreatedAt >= lastWeek && n.Priority > 1)
    .ToListAsync();

Paging, counts, existence checks:

var page = await Redb.Query<NoteProps>()
    .OrderByDescending(n => n.CreatedAt)
    .Skip(20).Take(20)
    .ToListAsync();

int total = await Redb.Query<NoteProps>().CountAsync();
bool any = await Redb.Query<NoteProps>().AnyAsync(n => n.Priority == 5);

When you do not need the whole object, take just the fields you want — less data off disk:

var titles = await Redb.Query<NoteProps>()
    .Where(n => n.Priority >= 3)
    .Select(n => new { n.Props.Title, n.Props.CreatedAt })
    .ToListAsync();

Worth noticing, because it catches everyone once: in Where and OrderBy you write properties
directly — n.Priority — while in Select you go through Props: n.Props.Title. In a condition
the parameter is your data; in a projection you get the whole object including service fields
(n.Id, n.name), so Props has to be explicit.

Arrays

An array property is not a delimited string — you can query it:

var home = await Redb.Query<NoteProps>()
    .Where(n => n.Tags!.Contains("home"))
    .ToListAsync();

Add a field

The most common thing you will do as the app grows. Add the property:

public class NoteProps
{
    // ...everything that was here
    public bool IsDone { get; set; }   // new
}

Done. The same InitializeAsync at startup picks it up. No migration files, no ALTER TABLE, and
objects saved earlier keep loading — the new property simply comes back as its default.

Compare that to the migration workflow: create a migration, review the generated SQL, think about the
rollback, ship it to devices and hope it lands cleanly on data you have never seen. That step just
does not exist here.

The outbox queue

This is usually the whole reason a local database exists, so here it is end to end. The job: while
there is no connection, changes pile up on the device; once it is back, they go out in order with a
status you can show.

The queue is a normal class. Note that it holds a typed payload with its own nested structure, not
a JSON string:

[RedbScheme("OutboxEntry")]
public class OutboxEntryProps
{
    public string Operation { get; set; } = "";      // "order.create", "order.update"
    public OutboxState State { get; set; }           // Pending, Sending, Failed, Sent
    public int Attempts { get; set; }
    public DateTime CreatedAt { get; set; }
    public DateTime? LastTriedAt { get; set; }
    public string? LastError { get; set; }

    public OrderProps? Payload { get; set; }         // the whole graph
}

Enqueueing is just a save:

await redb.SaveAsync(new RedbObject<OutboxEntryProps>
{
    name = $"outbox {order.Number}",
    Props = new OutboxEntryProps
    {
        Operation = "order.create",
        State = OutboxState.Pending,
        CreatedAt = DateTime.UtcNow,
        Payload = order          // the nested graph is stored with the entry
    }
});

Flushing once you are back online. This is where querying earns its keep: picking the right entries is
LINQ, not "load everything and filter in memory."

var batch = await redb.Query<OutboxEntryProps>()
    .Where(e => e.State == OutboxState.Pending && e.Attempts < 5)
    .OrderBy(e => e.CreatedAt)          // strictly in the order they appeared
    .Take(20)                           // in chunks, so you never stall
    .ToListAsync();

foreach (var entry in batch)
{
    try
    {
        await api.SendAsync(entry.Props.Operation, entry.Props.Payload!);
        entry.Props.State = OutboxState.Sent;
    }
    catch (Exception ex)
    {
        entry.Props.Attempts++;
        entry.Props.State = OutboxState.Failed;
        entry.Props.LastError = ex.Message;
    }

    entry.Props.LastTriedAt = DateTime.UtcNow;
}

// The whole batch in one call, not one write per entry.
await redb.SaveAsync(batch);

That last line matters: SaveAsync takes a collection and writes it as a batch. The loop keeps only
what is genuinely one-at-a-time — the network call — and the results go to the database in a single
trip. At twenty entries you will not notice. At two thousand you will.

Showing the user what is happening is a query too, not a tally in a loop:

int waiting = await redb.Query<OutboxEntryProps>()
    .Where(e => e.State == OutboxState.Pending)
    .CountAsync();

var problems = await redb.Query<OutboxEntryProps>()
    .Where(e => e.Attempts >= 5)
    .ToListAsync();

Try the same on "key → JSON": finding stuck entries means reading the entire queue, deserializing
every item and looping. And renaming a field means praying the old records still parse.

Making queries fast: filter on the base fields

A habit worth picking up on day one rather than when the list starts crawling.

Besides your Props, every object has the RedbObject's own fields: Id, ParentId, DateCreate,
plus fast slots — value_string, value_long, value_datetime and friends. They live right in the
object's row, which makes filtering on them the cheapest thing available: it narrows the set before
properties come into play.

You filter them with WhereRedb, which composes happily with regular Where.

The idea is simple: whatever you look things up by most often — a situation key, an external id, a
timestamp — put it in a fast slot as well as in Props. Then fetching state for a given key becomes
an indexed hit.

Writing — fill the slots along with the data:

var key = $"{stateType}:{documentId}";      // "order-draft:12345"

await redb.SaveAsync(new RedbObject<DraftStateProps>
{
    name = $"Draft {key}",
    value_string = key,                      // the situation key
    value_long = documentId,                 // external id
    value_datetime = DateTimeOffset.UtcNow,  // timestamp
    Props = new DraftStateProps { /* ... */ }
});

Reading — narrow by slot first, refine by properties after:

// state for a specific key — a hit, not a scan
var draft = await redb.Query<DraftStateProps>()
    .WhereRedb(o => o.ValueString == key)
    .FirstOrDefaultAsync();

// everything belonging to a document
var byDocument = await redb.Query<DraftStateProps>()
    .WhereRedb(o => o.ValueLong == documentId)
    .ToListAsync();

// cut by date: clean out what went stale
var threshold = DateTimeOffset.UtcNow.AddDays(-30);
var stale = await redb.Query<DraftStateProps>()
    .WhereRedb(o => o.ValueDatetime <= threshold)
    .ToListAsync();

// cheap slot filter first, property condition second
var actual = await redb.Query<OutboxEntryProps>()
    .WhereRedb(o => o.ValueDatetime > threshold)
    .Where(e => e.State == OutboxState.Pending)
    .ToListAsync();

Grouping is ParentId. Worth being precise here: this is not just a number, it is a real foreign
key to another object in the same database — to its Id. You cannot stash an identifier from some
other system in it; that is what value_long is for. What you get in return is database-level
integrity and cascade: delete the parent and the children go with it, no manual cleanup.

So the parent has to exist: save the session object (or document, or route) first, take its Id, then
put that Id into parent_id on the children. After which fetching the whole group is one condition:

var groupItems = await redb.Query<DraftStateProps>()
    .WhereRedb(o => o.ParentId == sessionId)
    .ToListAsync();

// or several groups at once
var manyGroups = await redb.Query<DraftStateProps>()
    .WhereRedb(o => o.ParentId != null && sessionIds.Contains(o.ParentId.Value))
    .ToListAsync();

Many-to-many without a join table

The same trick removes the reason you would normally introduce a junction table. Say you track
membership: users belong to groups, a document belongs to several categories.

Store the link as an object and fill two slots at once: ParentId for one side, value_long for the
other. Now both directions are a single indexed query with no JOIN:

await redb.SaveAsync(new RedbObject<MembershipProps>
{
    name = $"member {userId}",
    // one side of the link is the parent
    parent_id = groupId,
    // the other goes into a fast slot
    value_long = userId,
    // and into key: a unique index on it stops the same link
    // from being written twice
    key = userId,
    Props = new MembershipProps { AssignedAt = DateTimeOffset.UtcNow }
});

// everyone in the group
var members = await redb.Query<MembershipProps>()
    .WhereRedb(o => o.ParentId == groupId)
    .ToListAsync();

// every group a user belongs to — the reverse query, also JOIN-free
var groups = await redb.Query<MembershipProps>()
    .WhereRedb(o => o.ValueLong == userId)
    .ToListAsync();

This is not invented for the article: it is exactly how role assignments work in RedBase Identity,
where the link class carries a comment stating that parent_id points at the role, value_long
mirrors the user id for the reverse lookup, and key provides the unique index that makes assigning a
role idempotent without a separate existence check. Adopt the convention in your own classes early and
you will not be rewriting queries later.

Small gotcha: you write the lowercase names (value_string, parent_id, key) and read the
PascalCase ones in WhereRedb (o.ValueString, o.ParentId, o.Key). Same fields.

While we are here: trees

Since ParentId is a link to another object, objects naturally form a hierarchy — and you do not have
to hand-roll it. There is an API for it: load a whole subtree, take direct children only, build the
path to the root for breadcrumbs, move a node with everything under it, ask whether A is a descendant
of B, walk depth-first or breadth-first, select roots only, leaves only, or filter by depth.

// the whole branch in one query
var subtree = await redb.TreeQuery<CategoryProps>(rootId).ToListAsync();

// moving a node — children come along
await redb.MoveObjectAsync(node, newParent);

On a client that is usually a catalogue, a folder tree, an org chart or threaded comments — the stuff
you would otherwise assemble with hand-written recursive queries.

And there is quite a bit more

Not to turn this into a reference: beyond what is shown here, RedBase has aggregations and GroupBy,
window functions, lookup lists, polymorphic queries over a class hierarchy, soft delete with
background purge, built-in audit fields (who changed what, when), object ownership and permissions,
and database export/import. All of it behaves the same across all three providers — client-side SQLite
included.

The repository ships a redb.Examples project with 148 runnable examples grouped by topic:
queries, analytics, trees, lists, CRUD. Fastest way to see how a specific thing is done without
reading the docs end to end.

Syncing with a server that also runs RedBase

A pleasant side effect: RedBase is not only SQLite. The same classes run on a server over PostgreSQL
or SQL Server. Put the schemas in a shared project that both the client and the backend reference, and
the data model becomes literally one model for the whole system.

What changes in practice: there is no translation layer between client and server. No DTOs, no
mappers, no separate "sync contract" you have to edit on both sides every time a field changes. The
object you pulled out of the local database is the same type the server puts into its own:

// client: pulled from the queue
var entry = ...;

// server: received the same type and saved it
await redb.SaveAsync(order);

Add a field to the shared class and it appears in the local database, in the server database, and in
what goes over the wire. No keeping three places in sync, no matching a server migration to a client
version.

How it compares

Comparing in one specific role: private local storage for a client app. Not a server database, not an
analytics warehouse — the thing that sits on a user's device.

	EF Core + SQLite	Key → JSON	RedBase
Schema for a new kind of state	entity + migration	nothing	nothing
Hundreds of state classes	hundreds of tables and migrations	one table, no types	tag with `[RedbScheme]`
App update	migration runs on the user's device	—	nothing to migrate
Nested graph	`Include` / `ThenInclude` per level	all at once, as a blob	all at once, in one line
Querying nested fields	JOINs	not possible, memory scan only	LINQ down to SQL
Strong typing	yes	gone	yes
Raw SQL when you want it	yes	no	yes

The rows worth expanding on.

Migrations. On a server a migration is a controlled procedure: you apply it, watch it, roll it
back. On a client it ships to someone else's device and runs against data you have never seen. The
more your local state churns — and it churns more than server entities do — the more often you roll
those dice. That step is simply absent here: a new property appears on its own, old objects keep
loading.

The graph. With EF, completeness is your job on every single query: miss an Include and you get
an empty collection instead of data; add too many and you drag half the database into memory. On a
client, where you almost always want the whole graph (the user opened a draft), that is a daily tax.
LoadAsync returns the object assembled.

Typing vs JSON. The "key → JSON" route wins exactly once — on day one. After that: rename a field
and lose data silently; need a search and enjoy your full scan; want to eyeball what is actually
stored and good luck. RedBase gives you the same "just save the object" feeling, but the properties
sit in typed columns and take part in queries.

Raw SQL. Worth stating plainly, because the question comes up immediately: what if I want a flat
table for trends or aggregates? Nobody took it away — the same context runs arbitrary SQL, including
against your own tables:

@inject IRedbContext Context

var total = await Context.ExecuteScalarAsync<long>(
    "SELECT COUNT(*) FROM my_metrics WHERE bucket = '2026-08'");

await Context.ExecuteAsync(
    "CREATE TABLE IF NOT EXISTS my_metrics (bucket TEXT, value REAL)");

So it is not "objects or SQL" — it is objects by default, SQL where SQL fits better.

When sticking with EF Core makes more sense. If the app already has an EF model shared with the
server and there is no reason to rewrite it. Or if the local database has to have a specific physical
schema because something other than your app reads it. Outside of those, you are paying in migrations
and Include for a schema nobody outside will ever see.

What about MongoDB? There is no MongoDB on a client — it is a server. If what appeals to you is
the document model ("just save the object"), RedBase gives you that feeling on top of ordinary SQLite:
transactions, strong typing, and LINQ instead of a bespoke query language. Local document stores like
LiteDB are closer in spirit, but there you are back to choosing between "I store documents" and "I can
search."

Things worth knowing upfront

A few things that are better learned here than in production.

SQLite is single-writer. That is SQLite, not the wrapper. Rarely an issue for a client app, but if
you plan to write from several threads, keep transactions short.

One tab per database in the browser. Two tabs are two independent instances of your app, each with
its own copy of the file in memory. Last one to save wins. If you need multi-tab, coordinate through
BroadcastChannel or lock the second tab out.

The browser is single-threaded. A heavy query freezes the UI, so do not pull everything onto the
page: Take and paging are about responsiveness here, not aesthetics.

Download size. The managed assemblies come to roughly two megabytes, plus SQLite inside the
runtime. Fine for internal tools and offline-first apps, not for a landing page. Brotli is mandatory.

First launch in the browser takes a second or two while the database structure is created. Show a
spinner.

Exact money arithmetic. decimal is stored approximately in SQLite. If you need to-the-cent
precision, that is a SQLite constraint rather than a wrapper one — factor it into your choice of
storage.

Wrapping up

On mobile it comes down to three things: install the package, point it at a file in the app's data
directory, and call initialization once. After that it is ordinary C# with LINQ.

In the browser you add three more: wasm-tools with a build flag, the same manual initialization, and
your own IndexedDB persistence layer — where the things to remember are "move both database files" and
"never use the TRUNCATE checkpoint."

What you get for that is one data model across both clients, and queries instead of looping over
collections in memory.

Docs and examples: redbase.app. Sources, templates and the issue tracker:
github.com/redbase-app/redb.

More of my writing: dev.to/rinat_kozin.

If this was useful — a ⭐ on GitHub helps others find it.

A TMS backend, architecturally: cluster, coordinate streams and a typed object store

rinat kozin — Thu, 30 Jul 2026 22:07:45 +0000

This is a walkthrough of one system's architecture — no timelines, no budgets, no feature-matrix comparisons. Just how it is built: what the pieces are, how they spread across a cluster, where the data goes and why it goes that way.

The subject, briefly. It's a transport management system. Orders arrive from SAP; a driver and a vehicle get matched against requirements for both the vehicle and the stops. A trip runs through waypoints with time windows, checklists at each stop, and handover reports with damage records when a vehicle goes out and comes back. On top of that: a stream of GPS coordinates, tracking for trips and places, a mobile app for drivers, and a public zone where the customer watches their delivery.

Fifteen modules. Three application nodes behind a balancer, RabbitMQ and Kafka three nodes each, PostgreSQL, and Redis with six.

Standalone read — the earlier posts aren't required. If the economics interest you more than the wiring, they're covered separately: what this stack costs to keep and what changed when one module moved off the ESB. This one is only about the wiring.

What the stack is made of

Four levels, split on one principle: your own code lives only at the top, everything below arrives as a dependency.

A module is what you write in the project: route declarations, business rules, entities carrying schema attributes. Its entry point registers transport components, named connection factories and route builders. It ships as a package, lands in a directory, and the container picks it up live.

redb.Route is the integration engine — roughly forty patterns from the enterprise integration catalogue, 27 transports outward and four channels inward. It arrives as NuGet packages; if you need to go inside, you can reference the sources instead and step through with a debugger.

redb.Tsak is the module container and runtime: lifecycle, isolation, hot-swap, clustering with leader election, a dashboard and REST API, a scheduler, a route watchdog, a dead-letter queue. It deploys as a container image or a signed archive and is configured through environment variables.

redb is the typed store. Schema derives from classes; there are no migrations. Two doors: an object-level one for business entities, and direct SQL for what deliberately does not go into the object model.

The storage provider is chosen by the host — the module code never names it. PostgreSQL, MS SQL or SQLite, same schemas and same contracts.

The unit of composition is a context, not a service

Getting the scale right matters here. The deployment unit in this system is not a microservice and not a process — it's a module with its own route context.

A module is an assembly with an entry point that receives a context and fills it: transport components, named connection factories into a registry, services, sets of routes. The container then starts the context and runs the routes.

A context gives you three things at once:

an isolation boundary — each module gets its own assembly load context, so neighbours' dependencies don't collide;
a unit of distribution — contexts spread across cluster nodes and migrate between them;
a unit of control — a context can be stopped, started and restarted independently of the others.

Hence the practical consequence the whole design exists for: shipping one integration doesn't disturb the rest. The module package lands in a directory, the container picks it up and restarts only that context. Neighbours keep running; their in-flight messages survive.

Physical topology

Three application nodes behind a balancer. Node coordination runs through the shared database: leader election, route locks, redistribution of contexts when a node fails or is drained on purpose.

A word on responsibility boundaries, because one runs straight through here. The architecture and the application layer are mine. Deploying and operating the infrastructure is the DevOps engineers' territory: nodes, broker clusters, the database cluster, build pipelines, delivering secrets into the runtime.

That's not a formality, it's a working split. The architecture states requirements to the infrastructure — three Redis masters so a quorum exists; Kafka partitions so a consumer group can spread across nodes; database replicas. Meeting those requirements isn't my job. So what follows describes what the system needs and why, not how it's rolled out: that's someone else's work and someone else's decisions.

Each infrastructure component is clustered separately and on its own terms:

RabbitMQ — three nodes. Audit events, SAP integration, request-reply exchanges, outbound traffic. The public zone has its own separate broker; more on that below.

Kafka — three nodes. The GPS coordinate stream and the unplanned-stops topic.

PostgreSQL — three nodes, primary and replicas. It holds the object store, the partitioned tables, and the application cluster's coordination state.

Redis — six nodes. That one is worth explaining, because six isn't superstition.

Why Redis is exactly six

Redis cluster mode requires at least three masters. The reason is quorum: the decision that a master has failed and a replica should take over is made by a majority of masters. With two masters a majority is unreachable — one node cannot be a majority of two — so automatic failover is impossible in principle.

Three masters give a two-of-three majority. Each needs a replica, otherwise losing a master means losing its slots along with the data.

Three masters plus three replicas is six. Fewer than that and you either have no automatic failover or no data redundancy.

Two zones

The system is cut not only into modules but into zones with different trust levels.

The internal zone holds the system's verticals, the store, the brokers and the cache. Drivers work here through the mobile app, operators and administrators through the control panel — each with their own facade and their own permissions.

The public zone sits outside the company perimeter, in the cloud. It has its own database, its own broker, its own cache — nothing shared with the internal infrastructure. Customers work here: they look up where their delivery is.

And the key property: there are no return connections. The company backend reaches into the zone and writes data. From inside the zone outward is forbidden.

From which follows an architectural requirement that would otherwise look arbitrary: access validation inside the public zone must be local. You cannot ask the company whether a key is valid — no return connection exists. So the check is self-contained and runs on data already present in the zone.

Separately about the internal zone's external boundaries. The SAP bus is not our layer but someone else's system: orders, drivers, places and counterparties arrive from it. External routing is the source of planned geometry. Both stay put through any change inside — they're boundaries, not parts.

Transports and channels

The engine distinguishes two fundamentally different kinds of connection, and it pays to keep them apart on any diagram.

Outward — 27 transports. In catalogue terms that's the Channel Adapter: brokers (RabbitMQ, Kafka, AMQP, IBM MQ, Azure Service Bus, SQS, MQTT), protocols (HTTP, gRPC, TCP, WebSocket, SignalR), files (File, FTP, SFTP, S3), data (SQL, Redis, Elasticsearch), plus mail, LDAP, Telegram, a scheduler and process execution.

Inward — four channels, each solving a different problem:

Channel	Semantics	When you need it
`direct`	synchronous, inside a route	factor out a segment without changing control flow
`direct-vm`	synchronous, across contexts	call a neighbouring module with no network and no serialisation
`vm`	asynchronous, across contexts	decouple modules without standing up a broker
`seda`	asynchronous in-memory queue	separate processing stages with backpressure

The last one deserves a note. seda is a bounded queue with several competing consumers and an enqueue timeout. Need to decouple fast intake from slow processing? Put one between them and set a worker count. No broker required — and the queue being bounded gives you natural backpressure instead of unbounded growth in memory.

In catalogue terms that's a Point-to-Point Channel together with Competing Consumers: a pattern closed by a kernel primitive rather than external infrastructure.

The high-frequency stream: where the object model ends

Here the architecture makes a deliberate exception, and it deserves to be stated plainly.

Business entities — trips, waypoints, orders, drivers, vehicles — live in the object store, load as a graph in one call, and derive their schema from classes. But coordinates and audit events do not go there. They have their own road: partitioned tables and direct SQL.

The reason is simple: these are streams of a different order of frequency. The object model is optimised for a typed, linked graph; here you need bulk insertion of tens of thousands of uniform rows and later deletion by whole months. Different jobs, different tools — and forcing one onto the other ends badly for both.

One source, several aggregators

Coordinates arrive on Kafka. And this is where the interesting part starts: one stream carries several aggregation stages with different correlation keys and different completion rules.

Stage	Correlation key	Completion	Result
Persistence	trip	batch size or timeout	one bulk insert instead of a thousand singles
Unplanned stops	trip and vehicle	dwell time window	a stop event
Stop grouping	stop	end of a series	a coherent series instead of scattered points
Point deduplication	point	window	a stream without repeats

Plus point metrics, arrival and departure detection, and driver position broadcast.

That's the Aggregator pattern from the catalogue, applied several times with different settings. And it has one property that matters: a group completes on size or on timeout. Without the timeout, the last partial batch would sit in memory until the next message — which on a sparse stream means data not being persisted for hours.

How the stream scales across nodes

Kafka's own mechanism does the work here, and the code carries an explicit comment about it: do not pin a partition in the endpoint address, so the consumer-group mechanism engages.

The point is that Kafka itself distributes a topic's partitions among consumers sharing a group id. A node appears — partitions rebalance toward it. A node drops — its partitions spread across the survivors. No coordinator of your own to write.

In catalogue terms that's Competing Consumers, implemented at the broker rather than in the application.

The consequence everything else grows from

And now the crux. Partitions migrate between consumers. Contexts migrate between nodes. Which means coordinates for one and the same trip may be processed by any node in the cluster, and not necessarily the one that handled them a minute ago.

Hence a hard requirement: GPS processing state cannot live in a node's memory. It's shared, in Redis. Trip-point cache, accumulated processing state, planned and actual geometry. Anything needed to continue working on a trip has to be reachable from any node.

That's exactly why the system has a dedicated shared-cache manager rather than a dictionary in process memory.

Planned route and actual route

A trip has two geometries, and they're produced in completely different ways.

The planned one comes from external routing — how the trip is supposed to run. It's built once: on the first GPS point, or when an operator saves.

The actual one accumulates from the coordinate stream — how the trip really went. It has a lot of points, so it gets simplified with the Ramer–Douglas–Peucker algorithm: the line keeps its shape while the point count drops by an order of magnitude.

Both live in the shared cache, because both are needed by any node.

The broker carries a key, not geometry

Route geometry is a large object. Pushing it through a broker means loading the queues, inflating consumer memory and eventually hitting the message-size limit.

So it's done differently: the geometry stays in the store and the cache, and the message carries only a key. The consumer takes the key and reads the data itself — when it actually needs it.

In catalogue terms that's the Claim Check: a reference in the message, the payload stored separately. An old and well-known trick, which systems nonetheless keep reinventing after first getting burned by large messages.

Reads fall back through three levels

When a planned route is requested, the lookup walks a chain:

Redis — the cluster-wide cache. A hit answers immediately.
PostgreSQL — the geometry is persisted and survives cache eviction.
External routing — the last resort: a paid, slow call.

And the reverse chain: got it from the external service, save to the database, put in the cache, return. The next request for the same route is served from cache.

Net effect: the external call happens once per route, not once per request.

Audit: one event, two roads

Audit here isn't "logs just in case" but a proper layer with its own architecture. And it's built so that a single data change produces two independent flows.

Road one — into the store. An entity change is intercepted, the audit event lands in a queue, a background collector assembles a batch from it and performs a bulk insert into a partitioned table. The batch completes on size or on interval — both parameters live in configuration, not in code.

Note that this is the same pattern as on the coordinate stream. Different data, different frequency, different keys — one mechanism.

Road two — to subscribers. A copy of the event goes out through the broker to whoever needs it. Today that's at least two: GPS-zone cache refresh, and outbound exchange to external systems.

Crucially, the main flow does not wait for subscribers. The change persists at its own pace; the fan-out runs in parallel and its latency doesn't affect the write. In catalogue terms that's a Wire Tap plus a Publish-Subscribe Channel.

That construction delivers the thing audit layers are built for in the first place: any system can subscribe to changes without touching the one producing them. A new consumer of events shows up as a subscriber, and the source code doesn't change at all.

Partition maintenance

Partitioned tables need new sections created regularly and old ones dropped. That runs on a schedule, as scheduler jobs — and the scheduler is built into the container and cluster-aware, so the job doesn't fire three times on three nodes.

A detail people usually remember at the exact moment next month's partition failed to appear.

Observability

Once a system is spread over three nodes and several brokers, "where exactly is it slow" stops being answerable by reading logs.

Metrics are defined per pattern, not per engine. The aggregator reports groups completed, groups in flight and groups that fell apart on timeout. The splitter — how many parts. The filter — how many dropped. The idempotent receiver — how many passed and how many were rejected as duplicates. The circuit breaker — times tripped and calls rejected. And so on through the catalogue.

Plus end-to-end: processed, failed, currently in flight, and duration histograms for the whole exchange and for each route step.

Tracing follows OpenTelemetry, with the conventional attribute names: messaging system, destination name, operation, HTTP method, database type. Exceptions decompose into the standard fields. On top of those, its own: correlation id, exchange pattern, route, step, endpoint.

Practical consequence: trace collectors read this with no adapter. The chain "consumed from Kafka → aggregated → written as a batch → fanned out to subscribers" shows up as one trace with every step and duration.

The dashboard is a control plane, not a display

Eleven pages: overview, routes and single-route view, endpoints, cluster and node detail, scheduler, logs, audit, dead-letter queue, watchdog, access control.

And you act from them, not just look:

an individual route — stop and start it without touching the others;
a whole context — stop, start, restart;
a hung route — force-stop it;
a node — drain it gracefully: it finishes what it holds, takes no new work, hands its locks to peers; then bring it back;
a failed exchange — inspect and replay after a fix;
a scheduled job — fire it immediately;
a node's effective configuration — read it with secrets redacted, without SSH.

Next to each route are its own numbers: processed, currently in flight, throughput, history. Not one process-wide graph but per route — because in operations the question is always "which integration is stuck."

Where the boundaries run

More useful than any feature list is the set of places where the architecture deliberately departs from its own rules.

Partitioned tables bypass the object model. Coordinates and audit events are written with direct SQL. A conscious choice, not an omission: streams at that frequency have different economics.

Planned geometry is stored as a large object, not a graph of points. There's nothing to gain from decomposing it into entities — it's read whole and returned whole.

Processing state lives in Redis, not the store. It's hot, short-lived and needed by every node. Putting it in the database would mean paying for durability where none is required.

Access validation is duplicated locally in the public zone. Not for convenience, but because there are no return connections and nobody to ask.

The SAP bus and external routing stay external. They aren't rewritten or "integrated more deeply" — they're reached through channel adapters, and that's the correct boundary.

What's worth taking away

Strip the domain specifics and a few decisions transfer to any system of comparable shape.

Split flows by frequency, not by layer. Business entities and telemetry are fundamentally different loads. One storage mechanism for both usually means one of them is served badly.

An Aggregator isn't "accumulate and write" — it's a pattern with two completion conditions. Size and timeout, simultaneously. Without the second, a sparse stream simply stalls.

If state might be needed by any node, it isn't in memory. The moment a broker or a container starts redistributing work, local state turns into a source of hard-to-trace inconsistencies.

Large objects don't travel through the broker. A reference in the message, the payload in the store — and the queues stay light.

A fallback cache ends with a back-fill. Otherwise the expensive external call repeats on every request.

An event must be able to gain subscribers without editing its source. That's the property audit layers are built as layers for, rather than as a pile of log lines.

Transition status, so nothing is left unsaid: the public zone already runs on the stack described here; the remaining verticals are moving. A module-by-module before-and-after is in the second post; the economics of the "before" stack are in the first.

Sources and releases: github.com/redbase-app. About the redb store: redbase.app.

If this was useful — a ⭐ on GitHub helps others find it.

We ported a module off an ESB and the code grew 1.7 . That was the right outcome

rinat kozin — Thu, 30 Jul 2026 19:03:56 +0000

We're taking apart a transport management system — a TMS. Orders arrive from SAP, a driver and vehicle get matched against requirements, trips run through waypoints with time windows, checklists at each stop, handover reports with damage records. On top: a GPS stream into partitioned tables, trip and place tracking, a driver mobile app, and a public zone where customers watch their delivery. Fifteen modules, three nodes behind a load balancer, RabbitMQ and Kafka three nodes each, Redis, PostgreSQL, a separate identity server.

It was built because buying the TMS as a service from outside vendors got too risky — building it in-house was a way to get control of a core process back. And it was built fast: the system is sixteen months old, first commit April 2025.

In part one I took this system apart line by line and priced its infrastructure layer on WSO2 Micro Integrator and Entity Framework Core. It came to 3,700–6,800 hours one-off and 750–1,400 per year — 2.2–4 person-years up front and 0.45–0.85 of a permanent headcount annually, before a single line of business logic.

This post is what happens next. The system is leaving that state: the integration bus, the identity server and the ORM are all being removed.

I'll start with the module that has already moved — a line-by-line before-and-after. The headline result of that comparison was the opposite of what you expect from a migration post: there is now almost twice as much code. Why that's correct is the first half.

The second half is the other three layers, where the numbers run into person-years rather than hours.

Disclosure. I'm the architect and developer of the backend under analysis, and the author of the ecosystem it relies on. So this isn't a critique of someone else's decisions — it's my own bill. Which is also why measurements are kept apart from conclusions and numbers are broken out line by line: argue with the lines.

Scope: backend only. Web frontend is a separate team, mobile is a separate developer. Neither is in these numbers.

The team was complete: a systems analyst analysing requirements, a QA engineer verifying the work, a project manager running budget, deadlines, priorities, work order and risk. Worth remembering as you read — the accumulated infrastructure can't be explained away by "they had no process." And every hour quoted is development only.

Part of the redb / redb.Route series — recent posts first:

What an integration layer really costs: a TMS backend counted line by line — part one of this pair

A payment platform on .NET: what it actually costs

Leaving MassTransit for a Camel state of mind

redb.Route — Apache Camel for .NET

The timeline that explains everything else

Before counting anything, look at the dates. I pulled them out of the repository history.

When	What appeared
April 2025, day 1	first commit
April 2025, day 2	ORM with migrations
April 2025, day 8	in-house routing engine
June 2025, month 3	in-house module container
June 2026, month 15	first module on the new stack, no bus
July 2026, month 16	current state

Three observations, each worth stating.

The routing engine appeared on day eight. Not in year three after "hitting the limits of the platform" — immediately.

You do not write a 27,000-line routing framework in eight days. You bring it with you, finished. Which is where I should say plainly who and why: the architecture was designed before the project started, together with the tool that implements it. The engine wasn't shopped for on the market or picked from a comparison table.

What follows from that matters for reading the rest: the system is not accidental. Not one of the layers I take apart showed up by drift — each was chosen deliberately, and there's someone to answer for each choice. Everything called a cost below is my own bill, not a critique over someone's shoulder.

The bus stayed anyway. Both integration layers stood side by side from month one and still do. Not "bus first, then ours replacing it" — two runtimes simultaneously, all sixteen months.

Dismantling started in month fifteen. Not after five years of ossification, but almost as soon as the cost of what was chosen at the start became clear.

Which gives the conclusion I want at the top rather than the bottom. The infrastructure layer in this system is not an inheritance. It was created in sixteen months by decisions taken in the first week.

And that changes who this post is for. If you're reading someone else's tech-debt post thinking "my project is new, I don't have this" — here's a system sixteen months old with 105,000 lines of infrastructure in it. Tech debt doesn't accrue with time. It accrues with the stack you pick on day one.

First, how we count — because we count dishonestly

Before comparing anything, deal with the bias built into the counting method. It's in part one, it's in most migration posts, and I fell into it myself.

When we say "integration bus," we count our artefacts: 206 XML files, 4,099 lines of Java. We never count the platform's own internals — and that's a large Java product: runtime, mediation engine, dozens of transports, a management console. Hundreds of thousands of lines that somebody wrote, debugs and versions. Not one line of it enters the estimate.

When we say "identity server," we count 227 tables, because tables are visible in a database. The code serving those tables we don't count, though it exists and somebody maintains it.

And when it's our own engine, we count everything: 27,529 lines of routing engine, 11,117 lines of module container, and we invoice 1,940–3,400 hours for it.

The difference isn't economics. The difference is what happens to live in our repository. Someone else's internals are invisible, so they feel free. Ours are visible, so they feel like a line item.

This bias runs against the in-house stack, not for it

Let me state the counter-intuitive part directly: this way of counting inflates the price of your own solution and deflates the price of the bought one. Not the other way round.

Counted symmetrically, there are exactly two honest options.

Option one — count nobody's internals. Then the "free" bus is an external dependency you added. And an engine pulled in as a package is an external dependency you added. Both are zero in the budget. What you compare is what you wrote on top: 861 lines of XML plus 4,099 lines of Java, against 212 lines of route declarations.

Option two — count everyone's internals. Then next to 38,646 lines of engine and container stand the hundreds of thousands in the bus platform and the identity server. And the in-house engine turns out to be an order of magnitude smaller than what it replaces.

Both options give the same answer, and it's the opposite of the intuitive one. The asymmetric count — the one we apply by default — is the only one under which the in-house solution looks more expensive.

The part that isn't about line counts

There's one property that makes those two options unequal, though, and it's decisive.

The engine's internals never enter your repository at all.

It ships the way any normal platform ships: libraries as NuGet packages, the runtime as a container image or a signed standalone archive, releases on GitHub. You add a dependency and write your module. No sources in your tree, no building them in your pipeline, no upgrading them by hand.

Which is why those 38,646 lines are, in the new arrangement, not a cost line but zero lines in the repository. Exactly the sense in which a ready-made bus's internals were always zero. It's just symmetric now.

And on top of that there's something the vendor platform doesn't offer: if you need to go inside, you can. Reference the sources instead of the package and step through with a debugger all the way down. Not "fork it and live with the fork" — look, fix, offer it back.

The vendor platform gives you no such choice. There's a binary, and exactly one way in: fork it. Which is what happened in this project — the transport connector misbehaved, waiting for a vendor fix wasn't viable, and 2,429 lines of someone else's Java landed in the repository permanently, off the upgrade path.

That's the real difference, not the line count. By default you carry nothing — and if you need to, the door inside is open. A fork isn't a benefit of an open platform; it's the price of having no sane way in. Paid once, then again at every upgrade.

What goes and what stays

The layout from part one. The system had four infrastructure layers:

Layer	Cost	Fate
Integration bus: 195 artefacts of XML + Java	904–1,708 h	removed
ORM: 1,500 lines of configuration, 54 models, repositories	440–820 h	removed
Identity server: 4 databases, 227 service tables	part of 450–850 h	removed
Own routing engine and module container	1,940–3,400 h	stays, becomes the foundation

Three of four get dismantled. What remains is the one that cost more than the other three combined.

That's probably the main conclusion of both posts, and it isn't about technology. The cheap-at-the-door things left. The expensive-at-the-door thing stayed and keeps earning.

Where we started: the public zone

The first thing to move was the customer-facing zone — a public area pushed outside the company perimeter into the cloud.

   INSIDE THE COMPANY               CLOUD (public zone)
   ──────────────────               ───────────────────
     backend ──── writes ────────►  its own RabbitMQ
              (inbound only)              │
                                          ▼
                                    its own PostgreSQL
                                    its own Redis
                                          │
                                          ▼
                                    HTTP API → customer site

The key property: there are no return connections. The company backend reaches into the zone and writes data. From inside the zone outward is forbidden. Own database, own broker, own cache; the zone is self-sufficient.

Why start here becomes obvious if you think about risk. The boundary is already drawn architecturally, not sketched on a diagram. The zone can be replaced wholesale without touching anything inside the company: outward it talks only to the customer site, inward it doesn't call at all. The cost of a mistake is capped at one zone.

If you're planning a similar move — look for the same thing in your own system. Not the most important part, not the hardest. The most severed one.

One module, before and after

Before

Ten synapse-XML artefacts plus a Java mediator:

Artefact	Lines
Client API	345
Driver position proxy service	312
Delivery info proxy service	147
Error handling sequence	57
Redis configuration sequence	23
Token validation sequence	18
Two Redis connection entries	23
RabbitMQ endpoint	5
Mediator library descriptor	6
XML total	936
Java token validation mediator	183
Total across two languages	1,119

After

One C# module, eighteen files:

File	Lines
Client API routes	407
Driver position route	343
Delivery info route	289
Token validator	238
Configuration loader	172
Module entry point	120
Exception handler	83
API error model	46
Scheduled partition maintenance	42
Six typed configuration files	168
Two models and a validation result	33
Total	1,941

1,119 → 1,941. Growth of 1.7×.

This is usually where a migration post ends, because the number is inconvenient. Let's take it apart honestly — starting with method, because counting lines across languages is easy to rig in either direction.

How I counted

Three ways to count the same thing, all on the same code:

Method	Before	After	Growth
Raw lines, build files excluded	1,119	1,941	×1.73
Blank lines and comments excluded	921	1,330	×1.44
Build files included	1,582	1,971	×1.25

All three show growth, so it isn't an artefact of the method. I'll use the first row from here on, as the least flattering to the new code: if a conclusion survives that one, it survives the others.

The third row hides its own story. Packaging this module for the bus requires a 463-line build descriptor. The new module needs thirty. Fifteen times the size, in a file that does nothing but explain how to build everything else. I keep build files out of the main count to compare code with code — but dropping the fact entirely would be dishonest, because somebody maintains that too.

And the caveat without which the table is worthless: a line of XML and a line of C# are different units. Adding them into one metric is a rough volume approximation, not a complexity measurement. Which is why what follows takes apart the composition, not the total.

Why these two things are comparable at all

Let me justify the comparison itself, or the rest is a matter of taste.

Neither side is an arbitrary invention. Both implement the same industry catalogue — the one described in Gregor Hohpe and Bobby Woolf's Enterprise Integration Patterns, which formalises 65 messaging patterns: routers, splitters, aggregators, enrichers, translators and the rest. That's the industry's shared vocabulary. Apache Camel is built on it. So is WSO2 MI. So is the engine in this post.

Which means this isn't "someone's product versus my hobby project" — it's two implementations of one specification. The same pattern is a mediator tag on one side and a method call on the other. That's exactly why comparing declarations line for line is meaningful: both describe the same set of concepts.

Since we're here, let me check my own coverage claim against the code rather than the project page. And note the counting mistake I nearly made myself: the book is not just about routing. More than half the catalogue is channels, endpoints and message construction — implemented by transports and the kernel, not by the routing language.

Group in the book	What exists in the engine
Routing	Content-Based Router (`Choice`/`When`/`Otherwise`), Message Filter, Dynamic Router, Recipient List, Splitter, Aggregator, Resequencer, Scatter-Gather, Multicast
Transformation	Message Translator, Content Enricher (`Enrich`/`PollEnrich`), Content Filter, Normalizer, Envelope Wrapper (`Marshal`/`Unmarshal`), Claim Check, Datatype Channel (`OfType`)
Channels	Channel Adapter — 27 transports, Point-to-Point Channel (`direct`, `vm`, `direct-vm`, `seda`, queues), Publish-Subscribe Channel, Message Bus, Messaging Bridge, Dead Letter Channel, Invalid Message Channel, Guaranteed Delivery
Endpoints	Polling Consumer (`timer`, `cron`, file and FTP polling), Event-Driven Consumer, Messaging Gateway (`ProducerTemplate`), Competing Consumers, Message Dispatcher (`LoadBalance`), Selective Consumer, Idempotent Receiver, Transactional Client, Service Activator
Message construction	Request-Reply (`InOut`), Return Address (`ReplyTo`), Correlation Identifier, Message Expiration (TTL)
Management	Wire Tap, Message Store (`Replayable`), Message History (tracing), Control Bus, Process Manager (`Saga`)

That's around forty catalogue patterns, and not by stretching — each is backed by a DSL method, a transport, or a kernel primitive.

Channel Adapter deserves singling out, because the book treats it as central and real life proves it: connecting to things is the most laborious part of any integration. Twenty-seven transports: brokers (RabbitMQ, Kafka, AMQP, IBM MQ, Azure Service Bus, SQS, MQTT), protocols (HTTP, gRPC, TCP, WebSocket, SignalR), files (File, FTP, SFTP, S3), data (SQL, Redis, Elasticsearch), plus mail, LDAP, Telegram, a scheduler and process execution.

And second, the in-process channels where no broker is involved at all.

There are four, solving different problems. direct — a synchronous call inside a route. direct-vm — a synchronous call between different contexts in the same process; that's what module-to-identity-server traffic runs on, more on which below. vm — asynchronous handoff between contexts. And seda — an in-memory queue with a bounded size, N competing consumers, an enqueue timeout and queue-depth counters.

The value of that last one is that classic stage decoupling costs you no broker. Need to separate fast intake from slow processing? Put an in-memory queue between them and set a worker count. You don't stand up RabbitMQ to get an async stage inside one process — and the bounded queue gives you natural backpressure instead of unbounded growth.

In the book's terms that's Point-to-Point Channel plus Competing Consumers, and it's a case of a catalogue pattern being closed by a kernel primitive rather than external infrastructure. In the "before" stack the same decoupling meant a broker queue — with the connection, topology, monitoring and network hop that implies.

Every pattern is visible in telemetry

Here's what comes with that catalogue and usually gets lost in EIP discussions. The patterns aren't merely implemented — each one is instrumented separately, and not with home-grown statistics in a private format but with standard OpenTelemetry.

Metrics are defined per pattern, not per engine:

Pattern	What's measured
Aggregator	groups completed, groups in flight, groups timed out
Splitter	parts produced
Filter	messages dropped
Idempotent Receiver	passed, rejected as duplicates
Multicast	branch count, failed branches
Recipient List	recipients selected
Circuit Breaker	times tripped, calls rejected
Retry	attempts, successes after retry, exhaustions
Saga	completed, failed, compensated
Throttle / Timeout / Debounce	delayed, expired, discarded and flushed
Wire Tap	copies dispatched, copies failed
Dead Letter	messages dead-lettered

Plus end-to-end: processed, failed, currently in flight, and duration histograms — for the whole exchange and for each route step.

Tracing isn't home-grown either. Spans come from a standard activity source, and attributes follow OpenTelemetry semantic conventions — messaging.system, messaging.destination.name, messaging.operation, http.method, db.system, file.system, with exceptions broken into exception.type / message / stacktrace. On top of those, its own: correlation id, exchange pattern, route, step, endpoint.

The practical consequence: Jaeger, Grafana or Tempo read this with no adapter — subscribe to the source and you're done. The module container knows about telemetry and wires it up itself, so "consumed from queue → enriched → hit the database → replied" shows up as one trace with every step and duration.

The difference from the "before" stack isn't the presence of logs — everybody has logs. The difference is that "which step is slow and why" is answered by a trace, rather than by correlating journal entries from two runtimes by timestamp.

Beyond the catalogue there are nine more patterns absent from the 2003 book but now de-facto standard in Camel: Circuit Breaker, Throttle, Debounce, Delay, Loop, Threads, Sample, Validate, TryCatch.

And the project is moving fast. Twelve releases in three months — from 1.0.4 in early May to 3.4.0 in late July. In that window: concurrency handoff as a first-class .Threads(N) pattern, replay checkpoints for failed exchanges, an Amazon queue connector, and a shared runtime layer where patching the framework is a file swap with no rebuild. The catalogue keeps closing; transports keep landing.

Worth saying plainly in a post about migrating off a mature platform: the engine's youth is a real factor in the decision, and pretending otherwise would be dishonest. The flip side is equally real — what sat in a vendor's tracker for years here closes in weeks, because the author is in the room and priorities come from operations rather than someone else's roadmap.

The correction that matters: I was comparing the wrong things

Now I have to hand back my own number, because as stated it misleads.

Adding "all the XML" to "all the C#" is wrong, because each stack has two distinct layers: the declarative route description, and the imperative logic a declaration can't express. Count them separately or the multiplier means nothing.

Layer one — the route declaration. On the bus that's synapse-XML; in the new module it's a block of the embedded language. Comparing like with like:

Route declaration	XML	DSL	Density
Client API (three resources)	345	72	×4.8
Driver position	312	52	×6.0
Delivery info	147	53	×2.8
Error handling	57	35	×1.6
Total	861	212	×4.1

861 lines of XML against 212 lines of DSL. On the layer where comparison is actually valid, the new stack is four times denser. And that layer — "declarativeness" — is precisely what ESB posts usually name as the ready-made bus's main advantage. It lost fourfold.

Layer two — imperative logic. Both stacks need it: parse the payload, compare statuses, build the response. The difference is where it lives.

On the bus there's nowhere for it but Java. Here's how much Java sits in the bus project overall, not in one module:

Java in the bus project	Files	Lines
Forked transport connector — someone else's code, permanently maintained	12	2,429
Own class mediators — what the XML couldn't express	6	1,670
Total	18	4,099

Four thousand lines of Java hold up the "declarative" platform. Half a fork of someone else's connector; half in-house mediators written where XML ran out.

In the new module that layer didn't disappear — it stayed in the same file, in the same language as the declaration. That's the entire source of the headline growth: what used to be a separate project in a separate language with a separate build now sits next to the route.

Which restates the conclusion. Not "code grew 1.7× when leaving the ESB." Rather: the declaration shrank fourfold, and the imperative logic moved out of an invisible Java project into the file next to the route. The net increase is the price of that move from invisible to visible, and it's an honest price.

And second: the new code isn't written densely — deliberately

The new module is written in a plain, spread-out style, for the debugger. In practice that means:

every route step is a named private method with a doc comment — extract the data, resolve saved state from cache, resolve from the database, compare statuses, build the response. Not because there's no alternative, but so each step can take a breakpoint and show its state;
exchange property names are hoisted into thirteen constants instead of inline literals;
retry parameters are named constants of their own;
each step carries a numbered comment.

And crucially: the framework offers something this code didn't use. It has an expression subsystem with JSONPath, compiled and typed variants. The module extracts data by hand instead: parse the JSON, reach for fields one at a time, try/catch around it, scatter the results across six properties. Forty-odd lines where a handful of expressions would do.

How much tighter it could be, I haven't measured and won't quote. But several times over is realistic, and I'll assert that as the author of both sides of the comparison.

Now combine both corrections and the asymmetry becomes visible.

Synapse-XML has exactly one compression mechanism: move the logic into a Java mediator. That's how token validation became eighteen lines — not density, delegation. You cannot compress the XML without evicting logic from it; every mediation step is a tag.

The XML in this comparison sits at its floor — it gets no denser, and can only shrink by hiding. The new code sits above its floor by choice — for breakpoints, readability and line-by-line verification against the original.

So ×1.73 is not the price of the stack. It's the price of the style chosen for a one-to-one port. The same result on the same framework could have been several times denser — and would have lost exactly what the move was for.

Where the growth actually went

Before	Lines	After	Lines
Client API (XML)	345	Client API routes	407
Driver position (XML)	312	Driver position route	343
Delivery info (XML)	147	Delivery info route	289
Error handling (XML)	57	Exception handler	83
Token validation (XML + Java)	201	Token validator + result	255
Redis and RabbitMQ connections (XML)	51	Typed configuration	168
Mediator library descriptor (XML)	6	—	0
—	0	API error model, two domain models	62
—	0	Scheduled partition maintenance	42
—	0	Configuration loader, entry point	292
Total	1,119		1,941

Three sources of growth, none of them verbosity.

One: things that didn't exist before

Monthly partition maintenance on a schedule — forty-two lines that had no counterpart in the old version. Previously somebody else did it, or nobody did.

Configuration loader and module entry point — nearly three hundred lines. In the old arrangement the platform played that role: configuration arrived from its files, lifecycle was its business. Now it's explicit code you can read.

That's 334 lines of added, not rewritten. Subtract them and ×1.73 becomes ×1.44 — the exact figure you get counting without blanks and comments. Two independent methods landing on the same number is a decent check on both.

Two: configuration became typed

Fifty-one lines of XML connection descriptions became 168 lines of settings classes.

Sounds like a regression until you look at what you got. Before: factory names as strings, parameters mixed together, a typo in a parameter name discovered at startup at best and at first failure at worst. Now: typed classes, defaults, compile-time checking.

A hundred and seventeen extra lines is the price of a connection misconfiguration no longer being a production discovery.

Three: the hidden came out

This is the important one.

Token validation in the old version is eighteen lines of XML. Eighteen, because the actual work happened somewhere else: the XML only invoked a 183-line Java mediator. One piece of logic, two languages, two places to look for it.

Now it's 255 lines of C# in one file that opens in a debugger, takes tests, and turns up in a code search.

Same story with error handling and the delivery info route: what looked short in XML was short because of what had been moved out of it.

What actually changed

Three languages collapsed into one

Before: XML for routes, Java for what XML can't express, settings files for configuration. Three build pipelines, three debugging stories, three places to look for a cause.

After: one language, one pipeline, one debugger.

It's worth registering what that lost "line saving" actually meant. Eighteen lines of XML looked cheaper than two hundred lines of C# right up until somebody asked: where exactly does the validation happen? And answering required knowing that a Java mediator exists, where it lives, how it's wired, and who builds it.

Exactly one service is still our own

The new module has precisely one class with logic of its own — the token validator. The comment at the entry point explains why:

Only custom logic not covered by the DSL: the token validator. SQL, Redis and RabbitMQ run in the routes; there are no separate repositories.

Everything else — consuming from queues, writing to the database, cache access, the HTTP facade, partition maintenance — is expressed as routes. The HTTP facade is built with no controllers at all: three resources declared directly in routes.

The validator stayed custom not out of preference but out of necessity: the public zone cannot ask the company whether a key is valid — there are no return connections. Validation has to be local and self-contained.

Two details that say more than the numbers

The port is documented in a comment

The header of one new route says it outright: "one-to-one port of the XML proxy" — followed by the original's algorithm, step by step: what to pull from headers, what from the body, where to look up saved status, under what condition to update database and cache, what to return.

That's not documentation for its own sake. It's how you prove behaviour didn't drift: while the original algorithm is written down alongside, anyone can check.

Worth stealing for anyone rewriting something that works. Port one-to-one, pin the behaviour in a comment, improve later as a separate step. The temptation to "make it nice while we're here" is the main reason migrations fail.

The cache keys still remember the platform

Redis key names in the new module start with a prefix containing the name of the integration platform. The one that no longer exists in this zone.

It happened because the key format couldn't change — data had to stay compatible during the transition. The platform was switched off; its name stayed in the data structure.

Small, but telling: a migration doesn't end the day the old code is deleted. Traces linger in formats, names and schemas for a long time.

What the move cost

I was going to give a model with assumptions here, as in part one. I broke it into stages — study the original, port the routes, the token validator, configuration, testing — and got 193–335 hours, i.e. a week and a half to two weeks for the module.

Then I looked at the repository history.

When	What
30 June, 21:22	first commit of the module
1 July, 11:21	second
1 July, 12:25	third
7 and 9 July	two fixes from testing

The port took a day. The evening of 30 June and the first half of the next day — after that, only refinement from test results, two commits over nine days.

My model was off by roughly twenty times. I'm leaving that in the post exactly as it happened, because the discrepancy is more interesting than the estimate: an expert estimate built on familiar yardsticks missed reality by more than an order of magnitude. Had I not gone into git, I'd have published plausible and entirely wrong numbers.

Why a day and not two weeks

Three reasons, and the first is one that migration posts don't yet discuss.

The port was done in a pair with a language model. And it worked not because "LLMs can write code now" but for a specific reason: the engine's DSL follows Apache Camel's idiom — the same catalogue patterns, the same from → process → to shape, the same URI endpoint addressing.

And language models know Camel very well. Twenty years in production, thousands of projects, mountains of examples, docs and forum questions — all of it in the training data. Which means following an industry standard had an unexpected side effect: the model already knows your API even though it has never seen it. It recognises the idiom.

That's worth absorbing if you design libraries today. The old argument for a familiar idiom was that developers onboard faster. There's now a second one, and it's heavier: the tool the developer writes with knows it too. Your own original idiom forfeits that — and the price of originality just went up.

I checked the reverse on the same task: synapse-XML with platform-specific extensions and artefacts referencing each other by name takes assistance noticeably worse. The format is narrow and there's little of it in the world.

Second reason — a one-to-one port. The original algorithm is known; there's nothing to invent. The task reduces to translating between two description languages, which is exactly what models do best.

Third — the same person wrote both sides. Both the source system and the target engine. That's an advantage nobody else has, and crediting the tool for what's explained by domain knowledge would be dishonest.

And deploying it — an hour by hand, fifteen minutes with a model

Second number in the same series, and it maps straight onto part one.

The "before" estimate has a line: "deploying and configuring the bus, clustering, training the team — 300–500 hours." That isn't padding: standing up a bus, clustering it, configuring the configuration stores and teaching a team to live with it is a couple of months of work.

There's something to compare against. Take the deployment of another system on the same stack: the topology is described in a single container compose file, and here's what comes up from it:

three worker nodes in a cluster, coordinating through a shared database, one group name;
a management web console across all three, reaching them with a service key;
a search engine with its UI, and object storage;
the application web frontend.

Plus the cross-cutting configuration: database connection string, queue factory parameters in the default context inherited by every module, authentication with signing and roles, secrets externalised to environment variables.

That file was assembled with a language model in about a quarter of an hour. By hand, without an assistant, around an hour: it's a ten-kilobyte file and it needs writing carefully.

Both numbers are honest and both are worth holding: an hour if you write it yourself, fifteen minutes paired with a model. Against three hundred hours the difference between them stops mattering, but claiming fifteen minutes as the general result would be sleight of hand.

The reason a model is effective here is exactly the reason it was effective on the routes: there is no proprietary deployment format. An ordinary container compose file plus configuration through environment variables on a predictable scheme. The model has seen thousands of those and writes them confidently.

Contrast with the alternative: the platform has its own deployment descriptor, its own build artefact, its own configuration wizards — and that a model won't write, because examples are scarce and the specifics are vendor-bound. There you're alone with the vendor's documentation, same as ten years ago.

And shipping the modules themselves is a file copy. Built packages go into a directory mounted into the container, which picks them up on the fly. No image rebuild, no process restart, no deploying the whole system for one integration.

The level isn't "development-grade" either. Everything that usually makes up those three hundred hours arrives pre-assembled:

three operating modes — standalone process with no database, single node with storage, or a cluster with leader election and automatic context redistribution;
observability — process and per-route metrics, ring-buffer logs, OpenTelemetry traces, optional metrics scraping;
a watchdog — detects suspect and hung routes and can restart them itself;
a scheduler — schema auto-created on first start, cluster-safe;
security — signed API keys, roles, expiry, revocation, user binding;
module isolation — each gets its own load context; dependencies don't collide;
management — REST API, CLI and a web dashboard, in the box rather than "write your own."

Nothing here is assembled piecemeal. A module goes into a directory and the container picks it up live, without stopping its neighbours.

And separately about the dashboard, because it's usually the thing people pay a commercial bus for.

Eleven pages: overview, routes and single-route view, endpoints, cluster and node detail, scheduler, logs, audit, dead-letter queue, watchdog, access control.

And it's a control plane, not a viewer — a distinction worth stating.

An individual route can be stopped and started from the dashboard, leaving the others alone. A whole context too: stop, start, restart. A hung route can be force-stopped. So the answer to "turn off that integration, it's taking down the neighbouring system" is two clicks, not a redeploy and not a ticket to DevOps.

Next to each route are its own numbers: processed, currently in flight, throughput, history. Not one process-wide graph but per route, because the question is always "which integration is stuck," not "how is the service feeling."

The watchdog gets its own page: suspect and hung routes, automatic restart, toggled right there.

Cluster handling isn't just a node list. You can see who the current leader is and when each node last reported. A node can be drained gracefully: it finishes what it's holding, takes no new work, hands its locks to peers — then comes back. You can rebalance everything, or evict a node hard. Node detail shows CPU, memory, threads, GC with history, its contexts and its logs.

Also from the dashboard: failed exchanges can be inspected and replayed after a fix, a scheduled job fired immediately, and a node's effective configuration read with secrets redacted — without SSH.

Line that up against the real failure modes from part one: consumers that silently stopped reading a queue, a hung request-reply exchange, manual restarts after a cluster rebuild. Every one of them is precisely the case this dashboard exists for — and each was being handled by walking onto a node instead.

This is where the DevOps person-hours actually are. Not in shorter routes, but in a management layer that doesn't take weeks to stand up and isn't hand-written for the third time this decade.

The limits of that number

A day is the port, not the whole thing. Outside it: studying the original, done earlier and separately; testing and behaviour verification against the old zone; two fixes from test results. Full elapsed time from first commit to last is nine days.

And to repeat the third reason, because it's decisive: going from two weeks to one day is not a property of the tool but the sum of three circumstances, of which two are reproducible. If you have a one-to-one port and a framework with a standard idiom, the first two are available to you. Both sides of the system in one head, probably not.

What it saves per year

The saving isn't in lines, it's in what stops happening:

What goes away	Hours/year
Maintaining XML artefacts for this zone	15–25
Diagnosing request-reply hangs on a shared channel	20–40
Manual restarts after cluster rebuilds	10–20
Maintaining the Java mediator: separate pipeline, separate competence	15–30
Total	60–115

Payback measured against the fact rather than my failed estimate: a day of porting against 60–115 hours saved per year — the investment returns inside the first month.

What isn't in the calculation is what can't be counted: silent failures. A consumer that stopped reading and didn't say so. An exchange hung on a shared channel. A night-time restart after a routine upgrade. Part one covered those in detail — each costs more than any row in a table, and none can be estimated in advance.

The real reason for the move isn't hours. It's that failures became visible and diagnosable.

Two runtimes on one node

Only one module has moved so far. But something important is already visible if you look not at code but at where all this physically runs.

The cluster topology, per the system's architecture documentation: three nodes, and each of them runs the in-house module container and WSO2 MI side by side. Not "the bus lives in some separate zone" — two different runtimes shoulder to shoulder, on the same hardware, for the whole life of the project.

That's the real cost of a heterogeneous stack — not the abstract "two build pipelines" line from part one, but literally: each of three nodes needs two runtimes serviced with different tools from scratch.

WSO2 MI is packaged into a Carbon Application (.car) via Maven — its own build pipeline, its own artefact versioning, its own rollout process per node.
A module on the new container is an ordinary .NET assembly dropped into a container volume. Which modules load on a given node is one environment variable with wildcard patterns — no image repackaging, no separate release per module.

For a DevOps engineer running three nodes, that isn't "a bit more work" — it's a second full area of expertise alongside the first: Java, Maven, Carbon packaging, its own logs and metrics. And all of it is needed as long as a single node in the cluster still runs the bus, no matter how many modules have moved.

That's the key non-linearity of this transition's economics. Savings on DevOps competence don't scale with the number of migrated modules — they land in one step, when the last Carbon Application comes off the last node and the cluster stops being a two-runtime cluster. Until then you carry the second competence in full, not "half, since half the modules moved."

Layer two: the ORM whose migrations disappear

Analysis of the application layer produced this. All measured, not estimated:

	Count
`DbSet` declarations	51
Lines of model configuration in one method	~1,500
Files with eager-loading chains	70 (40 + 30 nested)
Files calling `SaveChanges`	40
ORM models with navigation properties	54
SQL views assembling graphs into JSON	5, of which one is used, four dead
Generic repository + specifications	yes
Migration files	3

And the question all of it was counted for: what here genuinely requires an ORM?

By the code: two partitioned tables, five views and three raw SQL calls. Everything else is servicing the ORM itself.

Fifteen hundred lines of model description and a migration system exist to serve two partitioned entities and three queries. Partitions and raw SQL need no ORM — they work over a direct connection.

The five views that were a workaround

They assemble object graphs into JSON — doing on the database side what the ORM couldn't do acceptably. Four of five are used by nobody: dead code nobody dared delete because it's unclear who might be watching.

An object store assembles the graph in one call natively. The views don't get "rewritten" — they stop being necessary.

What losing migrations means — not for the developer, for the business

This is the part technical posts mention in passing, and it's the most expensive one.

A migration isn't a repository artefact. It's an operational event. Every migration against a live system means:

a release window agreed with the business;
a rollback plan, and an answer for what happens if rollback fails;
someone with production database rights;
three nodes where application order matters;
a freeze on parallel schema changes in other branches until this one lands.

In an object store the schema derives from types: add a property to a class and it exists at the next initialisation. No migration file, no window, no rollback, no person with rights.

That isn't "more convenient for developers." That's an entire class of operational events disappearing from the company calendar — along with the approvals, the night windows, and the releases pushed because a migration didn't apply on the second node.

The measured size of what gets scrapped

I counted the data layer whole, separating human-written from tool-generated — they can't be mixed:

What goes	Files	Lines
Data layer: 55 models, reference data, interceptors, services	88	7,547
Database provider: a 2,049-line context, key generator, migration bodies	8	5,125
Written by humans	96	12,672
Generated by tooling: model snapshots	4	17,407
Total leaving	100	30,079

Thirty thousand lines. A hundred files. But the interesting part is the second-to-last row.

Generated model snapshots — 17,407 lines across four files, four thousand-odd each. Schema descriptions at each migration point plus the current snapshot. The tooling writes them itself.

Those files live in version control. They show up in pull requests. They conflict on branch merges — and nobody wants to resolve a conflict inside a four-thousand-line generated snapshot. Nobody reads them, everybody has to commit them.

Seventeen thousand lines no human wrote and no human reads, versioned, reviewed and conflicting like ordinary code. That's what "a migration system" physically means in a repository, on top of the night windows in production.

Plus 2,877 lines of hand-written migration bodies — those were written by hand, 2,748 of them in a single schema-creation file.

With a schema derived from types, neither exists. Not "there's less of them" — there are none.

Layer three: integrations collapse — but not on their own

Here's the causal link worth not losing. Integrations get simpler not because they were rewritten better. They get simpler because the ORM left them.

There are four integration modules: transforming data from SAP, persisting the result, outbound exchange via outbox, and a host application.

The first is pure transformation and never touches the database. The ORM never touched it, and it won't change at all.

All the bulk sits in the second. What it does today, verbatim from the code: look up an entity by external identifier, create or update a graph of related objects, manually synchronise collections, save changes. Once per entity type arriving from SAP — and each drags two to four related objects behind it.

What remains with an object store: save the object in one call. The graph goes as a whole.

What the integration did	What it becomes
Lookup by external key → create/update graph → save changes	Query by external key → save object
Manual synchronisation of nested collections	Part of the graph; no separate code
Custom reference-data cache on a dictionary	Built-in list provider with TTL
Reading an external identifier via a separate no-tracking context	An ordinary query

Why code shrinks here and grew in the bus module

This is the important reversal, and it explains both numbers at once.

In the bus case the code grew 1.7× — because logic hidden inside a Java mediator came out into the open. Volume rose; nothing stayed concealed.

In the ORM case the code shrinks — because nothing comes out into the open here. What disappears is ceremony: eager-loading chains, manual object assembly from separate queries, a generic repository with specifications. That isn't business logic you can lose. It's the overhead of talking to an ORM.

The analysis author's estimate for the three heaviest spots — flagged as an estimate, not a measurement: an 810-line processor spending roughly 60% of its code on loading and manual graph assembly compresses to about 300; another at 495 lines with six separate queries to about 150; a 163-line filtering service to about 60.

Fewer lines here isn't the goal, it's the consequence. The goal is that six database round-trips to assemble one object become one. And that is about hardware: fewer queries for the same answer means less load on the same nodes.

Layer four: identity on someone else's stack, and HTTP where a call would do

Now the architecturally interesting part — and probably the most underrated in money.

What exists today: an external identity server. Four separate databases, 227 service tables — against the 53 that hold the actual business of this system. Roughly four to one in favour of authentication infrastructure.

Nine integration points with the bus are confirmed in the code: token issue, refresh, revoke, JWT decode and validation, password recovery and reset, user binding, group sync, adding a user to a group.

And here's the detail worth stopping on. Among the confirmed scenarios is service-account token retrieval for two channels — mobile and the control panel.

A service account is machine-to-machine. No user, no browser, no redirect. One part of the system proving to another that it is what it says it is.

The path of that request, per the architecture documentation, runs through several layers: client → bus → identity server → backend.

So to get a token that a machine needs to talk to a machine inside one perimeter, the system goes over HTTP through two intermediate layers. With serialisation, TLS establishment, timeouts, retries, and the diagnostics of "which of the four hops just failed."

What changes with an in-house identity server

In the in-house solution every entry point is registered on an internal transport that works in-process, without the network — synchronously, zero-copy, between modules of one runtime.

A module needing a service-account token gets it with a method call. No HTTP listener, no TLS handshake, no JSON over loopback. The exchange goes from the route straight into the handler and back, on the same thread.

The principled split that follows:

Scenario	Needs a browser	Works over any transport
Service account (machine-to-machine)	—	✅
Redirect to the sign-in page	✅	—
Exchanging a code for a token	—	✅
Refresh, revoke, introspect, userinfo	—	✅
Management, user directory, audit	—	✅

A browser is fundamentally required by exactly two interactions in the whole standard: the authorisation redirect, and the verification page for input-constrained devices. Everything else is transport-neutral.

Hence the main conclusion for this layer. The network between our own services appeared not because the protocol needs it, but because the identity server was someone else's, and HTTP was the only way to talk to it. Once the stack stops being someone else's, the network on that hop disappears as a category.

Plus 227 service tables of a foreign schema are replaced by two dozen typed schemas derived from classes. With no migrations, for the same reason as the previous section.

The system isn't sliced only in layers — and that multiplies everything

So far I've talked about horizontal layers: facade, integrations, data, identity. But that's how the system looks only on a presentation slide.

In reality a vertical slicing runs across the horizontal layers: control panel, mobile channel, GPS zone, geometry, SAP integration, outbound exchange, reference data, scheduling, key issuance. Fifteen module entry points in the codebase.

And here's what matters: each vertical drags every horizontal layer through itself. Its own facade, its own data access, its own models, its own database context wiring. An abstract context, a provider context, a context factory, a provider selection service, a design-time factory — all of that exists once per system, but every vertical is threaded into it.

So thirty thousand lines of data layer isn't "one layer out of four." It's what cuts across all fifteen verticals at once. Removing the ORM isn't extracting a module — it's pulling out the wiring that runs through every module.

Adding up what's measured — what leaves the repository:

What	Files	Lines
Bus artefacts in XML plus build descriptors	213	17,799
Java: connector fork and class mediators	18	4,099
Data layer and database provider — written by humans	96	12,672
Routing engine and module container — not deleted, externalised as a dependency	227	38,646
Total written code leaving the repository	554	≈ 73,200
On top: generated migration artefacts	4	17,407

Against roughly 175,200 lines written by humans. About forty-two percent of the codebase stops being something this team maintains — or about forty-seven counting the generated artefacts.

A word on those percentages, because they're easy to spin. I deliberately keep human-written and generated in separate rows: mixing the bases would be double counting. And "forty-two percent" is not "we deleted forty-two percent of the system." It's the share of code this team no longer answers for in this repository: some genuinely deleted, some turned into an external dependency.

What this does to development speed

Here's the cost line nobody puts in a budget and everybody pays.

When one vertical needs a change, how many competences do you call in? In the "before" stack: routing XML, Java for what XML can't express, Maven and artefact packaging, an ORM with migrations, a foreign identity schema of 227 tables. Even if one person knows all of it, they're switching between five contexts; if they don't, the task travels between people with losses at each seam.

In the "after" stack — C#. One. Routes, logic, data access, configuration, tests, debugging. One language, one pipeline, one debugger, one breakpoint through the whole chain from HTTP to database.

That changes not just the hourly rate but the time to enter a task. A new vertical can be written off the cuff: take the neighbouring one as a template, repeat the structure, change the content. Less up-front analysis is needed, because there are fewer decisions of the form "which language does this part live in" and "where will it run."

And separately — the thing enterprise-migration posts don't discuss yet, though it's already part of the working day. Uniform code in a mainstream language with recognisable patterns is written several times faster paired with a language model than heterogeneous code is. The reasons are prosaic: the model knows the mainstream language and the standard patterns; repeated module structure means a template sits right next door; typing and the compiler catch mistakes immediately rather than at runtime on the third node. Each of those cuts both time and the amount of context you have to hold — in your head or in the tool.

The reverse holds too, and you find out on the first attempt: synapse-XML with platform-specific extensions and name-based artefact links takes that help noticeably worse. Not because the format is bad, but because it's narrow and rare.

So where's the business benefit

All four layers, along the axes a business actually measures.

What you don't write and don't carry. The routing engine, the module container with dashboard and clustering, the identity server, the data access layer — none of it gets written in the project and none of it lands in the repository. Libraries arrive as packages, the runtime as an image or a signed archive. What stays yours is route declarations, business rules and entities. In the module we took apart: one class with custom logic in the whole module, everything else routes and configuration.

Hardware. One runtime per node instead of two side by side. One database round-trip to assemble an object instead of six. The network between our own services on the identity hop disappears entirely — four network hops become a method call.

Competences. Before: routing XML, Java for the gaps, Maven and bus artefact packaging, an ORM with migrations, a foreign 227-table identity schema. After: one language, one pipeline, one debugger. And — more importantly — that saving arrives in one step, when the last artefact comes off the last node, not proportionally to migrated modules.

Development hours. Where ORM ceremony leaves, code shrinks by multiples. Where logic emerges from hidden mediators, it grows — and that's the right kind of growth. Total volume is the wrong thing to watch. The right thing: about forty-two percent of the codebase stops being maintained here — some 73,200 lines of 175,200 written by humans, plus 17,407 lines of generated snapshots nobody reads but everybody commits.

Time to enter a task. One competence instead of five means a new vertical gets written off the cuff from a neighbouring template, with no prior decision about which language a part lives in. Uniform code in a mainstream language also pairs far better with a language model than a narrow declarative format with name-based links — and that's the working day now, not the future.

Operations hours. An entire class of operational events disappears: migrations with windows and rollbacks; manual restarts after cluster rebuilds; trips to a node to answer "what settings is it on"; picking apart last night's failure by hand in a broker.

Money on what no estimate covers. Silent failures. A consumer that quietly stopped reading a queue. An exchange hung on a shared channel. Four dead views nobody deletes because it's unclear who's watching them. None of these enter an estimate in advance, and each costs more than the ones that do.

The one number I won't give is what this is worth in currency. It depends on rates, team size, the local market for specific competences, and whether those competences sit in one pair of hands. I built an explicit model for the bus layer in part one; extending it across all four layers before three of them have happened would be selling an estimate as a measurement. What's measured here is code volume, table counts, integration points and topology. The conclusions from them are mine.

Honest limits

Three of the four layers haven't happened yet. This is the main caveat and it goes first. One module on the bus is done. Removing the ORM is a measured analysis of the codebase plus a three-phase pilot plan, not past tense: simple isolated entities first, then hierarchy, then the complex graph, and only then a decision about full removal. Moving to an in-house identity server is a chosen target. Everything named as a number is measured in current code; everything named as a consequence is a conclusion still to be proven by work.

One module isn't the whole system. The zone that moved had its boundary drawn from the start. Modules wired into internal zones, SAP and the identity server move harder, and these numbers don't extrapolate linearly.

The author is the architect and developer of this backend and the author of the stack it moves to. Stated at the top, repeated here: the store, the routing engine, the container and the identity server are mine. That's simultaneously the source of numbers nobody else has, and the reason to read the conclusions critically.

Code volume growth is real. ×1.73 raw, ×1.44 without blanks and comments. Some is explained by additions and typing, but not all: explicit code is inherently wordier than a declarative description. If volume matters to you, that's a price and you should know it.

The ORM comparison is incomplete on one axis. Some of the object store's query advantages — expression compilation to SQL, tree traversal via recursive CTE, change tracking — belong to the paid edition. The free one goes through JSON facets and is slower. If you're costing this move, pick the edition before you count, not after.

A storage paradigm change hits everything around the database. Data doesn't live in one table per entity. Anything looking at the schema from outside the application — reports, extracts, ETL, an analyst's hands in pgAdmin — will need adapting. The data stays in an ordinary relational database and is reachable by raw SQL, but familiar "select from the trips table" queries get rewritten. That's real work and it isn't in the estimates above.

ESBs keep their strengths. One administration point, out-of-the-box adapters, a model a team already fluent in it understands. If you have that competence and the failure modes from part one don't affect you, the move may not pay back.

An in-house engine isn't free. It cost 1,940–3,400 hours in part one. Those were spent earlier and elsewhere; a team without one doesn't start where this one started. Though that estimate is exactly what the accounting-asymmetry section at the top is about: it's computed under rules where a foreign platform's internals aren't counted at all.

I have no measurements on development speed. Not per vertical before and after, not for pairing with a language model. I consider the direction obvious and argue it from properties of the languages, not from a stopwatch. If you have numbers, they're more interesting than my reasoning.

Conclusion

The one-to-one comparison produced the opposite of what you'd expect: there is 1.7× more code (1,119 → 1,941 lines; 1.44× without blanks and comments).

But broken out by layer, that multiplier says the reverse of what it seems to. The route declaration shrank fourfold — 861 lines of XML against 212 of embedded language. What grew was the imperative logic, which in the bus lived as a separate Java project (4,099 lines across the whole project, 2,429 of them a fork of someone else's connector) and now sits in the same file as the route. On top of that, the new code is deliberately written spread out for the debugger and doesn't use the framework's own expression support — it could have been several times denser, whereas the XML was already at its floor.

What we got in exchange: three languages collapsed into one, all logic findable by code search, debuggable with an ordinary debugger, testable with ordinary tests. Exactly one class stayed custom — the one that had to.

And on the other three layers the picture inverts

Worth putting side by side, because the numbers went opposite ways and that isn't a contradiction.

Where logic was hidden — in Java mediators behind eighteen lines of XML — the code grew on porting. Hidden became visible, and volume honestly reflected it.

Where there was no logic at all but ceremony — fifteen hundred lines of model description for two partitioned tables, seventy files of eager-loading chains, a generic repository with specifications, five graph-assembly views of which four are dead — the code shrinks. There's nothing there to lose.

The difference between those two cases is the main practical takeaway of both posts. Before counting savings in lines, work out which one you're looking at: hidden logic, or overhead. In the first case volume will grow, and that's fine. In the second it will shrink — and that's not a reward for rewriting either, it's a sign you were paying to talk to your tooling.

And the largest saving isn't measured in lines on any of the four layers. It's the disappearance of operational events: migrations with windows and rollbacks, manual restarts after cluster rebuilds, four network hops where a machine asks a machine for a token inside one process, and a second area of expertise you must keep on staff as long as a single node still runs someone else's runtime.

If you've been down a similar road, post your ratio in the comments. Two things interest me: whether volume growth on the way from declarative description to explicit code is a rule or a quirk of our zone, and whether your numbers for code reduction after dropping an ORM match the estimate above. I'm happy to revise mine.

Sources and releases: github.com/redbase-app. About the redb store: redbase.app.

If this was useful — a ⭐ on GitHub helps others find it.

What an integration layer really costs: a TMS backend counted line by line

rinat kozin — Thu, 30 Jul 2026 18:56:57 +0000

Most cost-of-software posts are written about someone else's system, from the outside, by someone who never had to keep it running. This one is written from the inside.

We're taking apart a transport management system — a TMS. Orders arrive from SAP. A driver and a vehicle get matched against requirements for both the vehicle and the stops. A trip runs through waypoints with time windows, checklists at each stop, and handover reports with damage records when a vehicle goes out and comes back. On top of that: a GPS stream into partitioned tables, tracking for trips, places and counterparties, WiFi-based presence detection, a mobile app for drivers, and a public zone where the customer can see where their delivery is.

Fifteen modules. Three nodes behind a load balancer, RabbitMQ and Kafka three nodes each, Redis, PostgreSQL, a separate identity server.

Why it was built at all. The TMS used to be bought as a service from outside vendors, and the risk grew. When a core business process runs on an external service, risk converts directly into money: downtime you can't influence, fix timelines set by someone else's priorities, dependency on a single supplier when terms change. While the risk is small, buying beats building. When it grows, the arithmetic flips — and building your own becomes a way to get control back, not a way to save money.

So it was brought in-house.

And here's the thing worth saying immediately, because it breaks the usual frame of every tech-debt discussion. This system is sixteen months old. First commit April 2025, 931 commits so far.

This is not inherited 2000s legacy whose author left years ago. It's a young greenfield project. 175,000 lines of backend in sixteen months — and it has already accumulated an infrastructure layer that now has to be dismantled.

Disclosure. I'm the architect and developer of the backend under analysis, and also the author of the ecosystem it relies on. So what follows isn't a critique of someone else's decisions — it's my own bill. And that's exactly why measurements are kept separate from conclusions and the numbers are broken out line by line: argue with the lines, not with my good faith.

Scope: backend only. The web frontend is built by a separate team, the mobile app by a separate developer. Neither appears in these numbers by a single line. Everything below accumulated inside one server-side codebase.

Part of the redb / redb.Route series — recent posts first:

A payment platform on .NET: what it actually costs

Leaving MassTransit for a Camel state of mind

redb.Route — Apache Camel for .NET

About the team, because it changes how you read the numbers

The backend wasn't written solo, and it wasn't written in a vacuum. The team had:

a systems analyst — requirements arrived analysed, not paraphrased in chat;
a QA engineer — what got built was verified by someone other than its author;
a project manager — the full classic role: budget, deadlines, priorities, work order, risk assessment.

This matters in two ways.

First: the overhead counted below is not the result of a broken process. The easiest explanation for 105,000 lines of infrastructure is "they had no analysis, no testing, nobody minding the shop." That explanation doesn't work here. All of it was in place. The infrastructure layer didn't grow instead of process — it grew alongside it.

Second: every hour in the tables below is development only. Analyst, QA and management time isn't in there by a single hour, though the project wouldn't have shipped without them. Total cost of ownership is higher than my numbers, not lower.

What's actually in there

Counted from the repository, vendor libraries excluded.

	Files	Lines
C# — application layer, own engine, container	851	121,466
SQL — schema, migrations, reporting, reference data	28	17,341
synapse-XML — WSO2 MI artefacts	206	14,564
OpenAPI contracts	20	5,235
Razor — operator dashboard and host boilerplate	54	5,224
Java — bus connector and class mediators	18	4,099
Own JS and CSS	26	3,999
Maven — build descriptors for bus modules	7	3,235
Total written by humans	1,210	≈ 175,200
On top: generated migration artefacts	4	17,407

Those last two rows need explaining, because they usually get shovelled into one pile and the difference between them is the whole point.

Generated migration artefacts — 17,407 lines across four files. No human wrote them and no human reads them: they're model snapshots the ORM tooling produces by itself. But they live in version control, they show up in pull requests, and they conflict on merge. They're excluded from the "written" count deliberately — a separate row is more honest. We'll come back to them.

Build descriptors for the bus — 3,235 lines across seven modules, averaging 460 lines each. That's not routing logic, it's Maven configuration: how to package a module into an installable artefact. For comparison, the .NET build descriptor in the same system is 30 lines. A fifteen-fold difference in a file that does nothing except explain how to build everything else.

Six languages describing one system, nearly all with their own build pipeline, their own debugging story, their own release cycle.

Layer one: WSO2 MI and synapse-XML

Let's start with the layer that usually escapes cost accounting entirely, because people habitually file it under "configuration."

That's wrong. Synapse-XML is code. Declarative, narrow, limited in expressive power — but code: it has branching, error handling, transformations, external calls, and message flow control. The difference from C# isn't whether it's code. It's the length of the feedback loop: artefacts reference each other by name, expressions over the message evaluate at runtime, and a meaningful share of mistakes surface not at build time but in production.

One thing I should correct up front, because it's a common misconception — WSO2 MI does have a debugger. You can set breakpoints on mediators in Integration Studio and inspect the message context. "You debug it with logs" is false, and I wrote exactly that in the first edition of this post until someone corrected me. The tooling exists. What's expensive here is something else.

The deployment unit is not a file but a whole module: projects are packaged by Maven into a Carbon Application, a .car archive, and that's what ships to the bus. You can see it in the repo as <packaging>car</packaging> across six build descriptors.

195 artefacts, and what they cost

A model with explicit assumptions. Rates are per artefact, covering writing plus debugging — not typing speed. Adjust for your reality; the table is broken out so you can edit it row by row.

Artefact type	Count	Hours each	Hours
Sequences	96	4–8	384–768
Endpoints	42	2–4	84–168
Proxy services	10	6–12	60–120
APIs (routing + definition)	6	8–16	48–96
Scheduled tasks	4	3–6	12–24
Local entries, templates	12	1–2	12–24
Forking the Kafka connector: reading foreign code, fixing, extending	1	60–120	60–120
Java class mediators (1,670 lines)	5	25–50	125–250
Deploying and configuring the bus, clustering, training the team	—	300–500	300–500
One-off total			904–1,708

The Java sitting underneath

Artefacts aren't the end of it. Next to them sit 4,099 lines of Java across 18 files, and they split into two stories.

A fork of the official Kafka connector — 2,429 lines across 12 files. This one deserves telling in full, because it's more typical than it looks.

The vendor has a Kafka connector. It's in their repository, it installs normally, it's part of the box. The problem is that it didn't work properly: it had bugs.

Then comes the fork in the road familiar to anyone living on someone else's platform — wait for the vendor, or fix it yourself. Waiting wasn't an option, so the project forked the connector and fixed it in-house.

The price looks low in the moment: reading foreign code and patching it is cheaper than writing a transport from scratch. But the price arrives later and never leaves.

A fork takes you off the upgrade path. The version is pinned to a snapshot — an unpublished state of the vendor's branch. Every platform upgrade now means: take the new connector, re-apply your patches, verify nothing drifted. Or don't upgrade, and accumulate divergence.

And you now maintain Java you didn't design. Nearly two and a half thousand lines of someone else's architecture to re-learn every time something breaks.

This isn't about a bad vendor. It's a property of the model: when the platform is someone else's, any defect in it becomes your permanent line item — either you wait, or you fork and pay for the fork forever.

Class mediators — 1,670 lines. These are Java classes wired directly into XML sequences wherever the XML ran out of expressive power. Their list describes the boundaries of the language better than any argument:

Mediator	Lines	What it does
Image scaling	433	Resizing and thumbnails — binary data handling
OAuth token cache	402	State that outlives a single request
Unique key pool	401	Thread-safe block allocation of identifiers with auto-refill
Thread pool diagnostics	250	Runtime introspection
Token validation	183	Custom security logic

Every line is a place where a developer hit "XML can't do that" and switched languages. Binary data, cross-request state, concurrency, runtime introspection, non-trivial security — all outside what a declarative routing format was designed for.

The third row is worth a pause. A block-allocating identifier pool with auto-refill is the classic answer to "hitting the database for every key is too expensive." Here it had to be implemented as a Java mediator living inside the integration bus. The same mechanism shows up elsewhere in this story — but that's the next post.

Two real failure modes

Not hypotheticals. These are the ones that actually cost nights.

Sender and consumer on one connection — mutual deadlock.

The mechanics: request-reply means the sender, having posted a message, waits for the answer. If a consumer is sitting on the same RabbitMQ connection, that wait occupies the shared channel and everything stops. The exchange doesn't proceed; the connection hangs.

What it looks like from outside is worse: consumers silently stop reading the queue. They don't crash, don't log an error, don't raise an alert. You find out from the consequences — messages piling up.

The cure is separate connections: distinct factories for sending and receiving, identical parameters, declared separately. You can see it directly in the system's configuration — every factory has separate sender and listener blocks with identical settings. Duplication that exists purely to work around platform behaviour.

Temporary RPC queues don't survive a cluster rebuild. When the container nodes get rebuilt, the temporary reply queues on the bus-side producers don't come back. The usual fix is restarting the bus node — by hand.

Layer two: EF Core

Here's a picture familiar to anyone who has run a project on Entity Framework. With one caveat worth holding onto for this whole section: everything below accumulated in sixteen months, not over years.

	Count
`DbSet` declarations	51
Lines of model configuration in one method	~1,500
Files with eager-loading chains	70 (40 + 30 nested)
Files calling `SaveChanges`	40
ORM models with navigation properties	54
SQL views assembling object graphs into JSON	5, of which one is used, four are dead code
Generic repository + specifications	yes
Migration files	3

And now the question all that counting was for: what of this actually requires an ORM?

The answer from the code: two partitioned tables, five views, and three raw SQL calls. Everything else is servicing the ORM itself.

Fifteen hundred lines of model description and a migration system exist to serve two partitioned entities and three queries. Partitions and raw SQL require no ORM at all — they work through a direct connection.

The five views that were a workaround

The five SQL views deserve their own note. They assemble object graphs into JSON — that is, they do on the database side the work the ORM couldn't do acceptably. Four of the five are used by nobody: dead code that nobody dared delete, because it's unclear who might be looking at it.

Work	Hours
Modelling and configuration: 54 models, ~1,500 lines of mapping	120–220
Eager-loading chains and manual graph assembly across 70 files	180–340
Generic repository + specifications	40–80
Five SQL views as a workaround for graph loading	40–80
Migrations, snapshots, resolving their merge conflicts	60–100
One-off total	440–820

Layer three: what we wrote ourselves

Now the most interesting thing in this system, and the reason it's worth taking apart at all.

Running alongside WSO2 MI was a home-grown integration engine. Not a wrapper, not helpers — a full routing framework in the spirit of Apache Camel, written for .NET.

Component	Size
Routing engine	173 files, 27,529 lines
Module container with dashboard and REST API	54 files, 11,117 lines
Total	≈ 38,600 lines of infrastructure code

Inside the engine:

eight transports — Kafka, RabbitMQ, Redis, Cron, Timer, SEDA, Direct, Validator, each with its own consumer, producer and endpoint;
43 abstraction files — IComponent, IConsumer, IProducer, IEndpoint, IProducerTemplate: Camel's vocabulary, verbatim;
41 files of expression language — expressions and predicates, compiled rather than interpreted;
19 route definition files — i.e. its own DSL;
16 processors and 4 files of transaction handling.

And the container isn't "a host that starts modules." By inventory it has module lifecycle management, hot-reload, leader election, per-route metrics, a REST API and a web dashboard.

Work	Hours
Routing engine: transports, DSL, expressions, processors	1,200–2,100
Module container: lifecycle, hot-reload, clustering, dashboard, API	740–1,300
One-off total	1,940–3,400

That's 1.2–2 person-years, and the single most expensive line in the entire estimate.

The obvious question is why a system that already had an enterprise bus needed a second integration layer. The answer is in the previous section: because the bus covered the declarative part and stopped there. Everything past a branch went into Java, every missing transport went into a fork, every stateful thing went into a mediator.

Add up the workarounds and you find you've written half an engine anyway — just smeared across someone else's extension points.

Layer four: the identity server

Authentication runs on WSO2 Identity Server. It brings four databases that have to be created, seeded, versioned, backed up and upgraded along with the product.

227 service tables. For scale: the business schema of this TMS — trips, orders, drivers, vehicles, places, counterparties, tracking, everything — is 53 tables.

Four to one in favour of authentication infrastructure.

The bus talks to it through nine confirmed integration points: token issue, refresh, revoke, JWT decode and validation, password recovery and reset, user binding, group sync, adding a user to a group.

And one detail worth stopping on. Among the confirmed flows is service-account token retrieval for two channels — mobile and the control panel.

A service account is machine-to-machine. No user, no browser, no redirect. One part of the system proving to another that it is what it says it is.

The path of that request, per the architecture documentation, goes through several layers: client → bus → identity server → backend.

There's also a token cache implemented as a Java mediator inside the bus — 400 lines that wouldn't exist if the traffic were lower. The first validation goes to the identity server, which issued the token; after that, validation happens bus-side without touching it.

Work	Hours
Deployment, four databases, backup and upgrade procedures	150–300
Nine integration points on the bus side	120–250
Token cache as a Java mediator	60–120
User and group synchronisation	120–180
One-off total	450–850

What it costs every year

One-off is half the story. The other half arrives annually.

Two stacks instead of one

The application layer is .NET, the integration layer is Java with Maven. That's two build pipelines, two dependency sets, two upgrade cycles and two areas of expertise inside one team.

In practice it means "change how this integration behaves" needs a person who knows both — or a handoff between two people, with losses at the seam.

Recurring work	Hours/year
Maintaining synapse-XML artefacts	200–350
Keeping the connector fork in sync with upstream	60–120
Java mediators: separate pipeline, separate competence	80–150
ORM: migrations, snapshot conflicts, query tuning	150–300
Identity server: upgrades, four databases, certificates	120–250
Diagnosing silent failures (see the two failure modes above)	140–230
Annual total	750–1,400

The summary

Layer	One-off hours
WSO2 MI: 195 artefacts, Java, fork, deployment	904–1,708
EF Core: models, mapping, chains, migrations	440–820
Own engine and container	1,940–3,400
Identity server: databases, integration, caching	450–850
Total	3,734–6,778

Round it: 3,700–6,800 hours one-off.

That's 2.2–4 person-years — before a single line of business logic.

Annually: 750–1,400 hours, i.e. 0.45–0.85 of a permanent headcount, purely to keep the layer running.

The rate to apply is the fully loaded hourly cost to the company: salary, taxes, workspace, management, holidays and idle time. Not the offer number — that's the usual mistake, and it understates the result two- or threefold.

Infrastructure against application code

One more cut worth taking. Out of the 175,000 lines written by humans:

What it is	Lines
Own routing engine and module container	38,646
SQL: schema, migrations, reporting, reference data	17,341
WSO2 MI artefacts plus build descriptors	17,799
Contracts, dashboard, own JS and CSS	14,458
ORM configuration, models, repositories, workaround views	12,672
Java: connector fork and class mediators	4,099
Infrastructure total	≈ 105,000
Application code: rules, calculations, processing, module routes	≈ 70,000

Rules, calculations, validation, processing — the part anyone actually pays for — is about forty percent of what was written. The rest is data transport, configuration, plumbing and manoeuvres around tool limitations.

Roughly three to two against the application code: for every two lines doing the job, three lines exist to make doing the job possible. In my experience that's not an anomaly — it's the norm for enterprise systems on a ready-made integration platform. And those three lines are what this post is about.

Worth repeating: this ratio is inside a single backend. Web and mobile are other teams and aren't in the numbers.

What these numbers don't cover

The section without which the estimate would be dishonest.

This is a model, not a measurement. Nobody kept parallel time tracking per artefact. Hours-per-item is expert judgement and every row can be disputed separately. That's exactly why the tables are broken out rather than given as a total.

Some of this work would have been necessary anyway. Business logic — rules, calculations, validation — doesn't depend on the stack it's written in. It isn't in the numbers above and shouldn't be: only the infrastructure layer is counted.

Development hours only. Analysis, testing and project management aren't in the tables, though the team had all three. Total cost of ownership is higher than my figures — by how much depends on how your roles are split, so I'm not inventing a multiplier.

ESBs have real strengths. A ready-made bus gives you out-of-the-box adapters you'd otherwise write by hand, one uniform administration point, and a model a team already fluent in it understands. If you have that competence in-house, some of my estimates are too high for you.

The organisation could have bought some of this. Then the saving converts into licences and into not developing your own, rather than into person-hours.

And most importantly: the system works. It holds production, serves the SAP integration and the mobile clients — and, again, it's sixteen months old. In sixteen months the team stood up from scratch what used to be bought as a service. This isn't a post-mortem of a failure. It's an accounting of the price paid for a specific set of architectural decisions. The price may well be justified; the question is whether you know it.

What's next

Everything above is the "before" state. The system is leaving it: the integration bus, the identity server and the ORM are all being removed.

Three of the four layers get dismantled. The one that stays is the one that cost more than the other three combined.

The next post is about that transition — a module-for-module comparison, the same zone, the same job, bus artefacts and a Java mediator on the left, a rewritten module on the right. And the headline result of that comparison was the opposite of what you'd expect from a migration post: there is now more code, not less. Why that's the correct outcome is the next one.

If you have a similar stack and different numbers, post yours in the comments. I'm happy to revise mine — converging on a realistic model is more interesting than defending the first draft.

Sources and releases: github.com/redbase-app. About the redb store: redbase.app.

If this was useful — a ⭐ on GitHub helps others find it.

Reference architecture for a payment platform on .NET: the ledger, transaction boundaries, modules, and the cluster

rinat kozin — Tue, 28 Jul 2026 22:46:52 +0000

In the previous post I took a payment platform apart by cost: three layers, where the person-years go, what you can avoid writing. It left one question honestly open: fine, but how do you actually assemble it?

This is the answer. A reference architecture: the ledger model, transaction boundaries, module split, cluster topology, integration routes, and recovery points. With code and diagrams.

Caveat up front — this is a reference model, not a dump from one production system. The code shows the real API and patterns that work, but your accounting model will differ, and it should. More on that at the end.

The stack: redb, a typed store on PostgreSQL; redb.Route, an integration engine; redb.Tsak, a runtime; and redb.Identity, an OIDC/OAuth 2.1 server. All Pro, all free across the 3.x line.

Part of the redb / redb.Route series — recent posts first:

A payment platform on .NET: what it actually costs, and how much of it you never have to write

redb 3.4.0: day-two operations for a .NET stack

Leaving MassTransit for a Camel state of mind: the Kafka connector, Scatter-Gather, and transactions

Apache Camel for .NET, dissected: the HTTP connector with no ASP.NET MVC + the Content-Based Router

redb.Route — Apache Camel for .NET: 22 transports, 30+ EIP patterns, compiled DSL

Sources: github.com/redbase-app. Docs: redbase.app.

The whole thing on one screen

Top down first. Here's the platform in one picture; we'll take each block apart below.

                        OUTSIDE WORLD
   browser/mobile   acquirer     bank      partner   regulator
        │              │           │          │          │
     HTTPS          HTTPS      IBM MQ      SFTP      exports
        │              │           │          │          │
╔═══════▼══════════════▼═══════════▼══════════▼══════════▼═══════╗
║                      TSAK WORKER (clustered)                   ║
║                                                                ║
║  ┌──────────────┐  ┌───────────────────────────────────────┐   ║
║  │  identity    │  │            payments                   │   ║
║  │  .tpkg       │  │                                       │   ║
║  │              │  │  payments.Api     ← HTTP facade       │   ║
║  │  OAuth 2.1   │◄─┼─ payments.Acq     ← acquirers         │   ║
║  │  OIDC        │  │  payments.Bank    ← IBM MQ            │   ║
║  │  SCIM        │  │  payments.Files   ← SFTP registries   │   ║
║  │  audit       │  │  payments.Recon   ← reconciliation    │   ║
║  │              │  │  payments.Core    ← LEDGER + SCHEMAS  │   ║
║  └──────┬───────┘  └───────────────┬───────────────────────┘   ║
║         │   direct-vm://           │                           ║
║         │   (no network)           │                           ║
╚═════════╪══════════════════════════╪═══════════════════════════╝
          │                          │
     ┌────▼─────┐               ┌────▼────────┐
     │ identity │               │  payments   │
     │    DB    │               │     DB      │
     └──────────┘               └─────────────┘
       separate                   PostgreSQL
       named store                   (Pro)

Four things worth noticing immediately, because they drive everything else:

Identity runs in the same worker but on its own database. Payment modules reach it over direct-vm:// — an in-process transport, no socket and no TLS. From the outside it stays a normal OIDC server on HTTPS for browsers and mobile apps.

payments.Core has no external transport at all. It's pure ledger: schemas, posting rules, balance computation. Every entrance to it is direct-vm:// from a neighbouring module.

Each external protocol is its own module. The acquirer changed something and its module needs redeploying — the bank rail doesn't notice.

Many modules, one worker. Or several — but that's an operations decision, not an architectural one, and it's a config change. We'll come back to it in the cluster section.

The ledger: a posting that never changes

Start with the data model, because everything else follows from it — transaction boundaries, idempotency, reconciliation, and what you'll be able to tell a regulator three years from now.

Decision #1: postings are append-only

The single most important architectural decision fits in one sentence: a posting object is created once and never modified. No status field, no cancellation, no amount correction. Made a mistake? Write a reversing posting; don't edit the old one.

Why this matters specifically here: redb has no built-in object versioning. If you update postings, you'll be building the change history yourself. If you don't — the history is the posting journal, and there's nothing to build.

That's not an engine limitation, that's correct accounting. Bookkeeping has worked this way for two hundred years.

[RedbScheme(Name = "payments.posting", Alias = "Posting")]
public class PostingProps
{
    /// <summary>Business operation key. One operation → several postings sharing it.</summary>
    public string OperationId { get; set; } = "";

    /// <summary>Account. A flat id, not a reference — see below.</summary>
    public long AccountId { get; set; }

    /// <summary>Signed delta: positive is a credit, negative is a debit.</summary>
    public decimal Delta { get; set; }

    public RedbListItem? Currency { get; set; }
    public RedbListItem? Kind { get; set; }

    public DateTimeOffset OccurredAt { get; set; }

    /// <summary>Set only on reversals — points at the posting being reversed.</summary>
    public long? ReversesPostingId { get; set; }

    public string? Reference { get; set; }
}

Three details, each a deliberate choice.

decimal Delta lands in a NUMERIC(38,18) column. Nothing to configure: any decimal in props maps to a column with 38 digits of precision, 18 of them after the point. Fees, FX and tax compute without accumulating rounding error — and eighteen decimals happens to be exactly the precision Ether is denominated in, if a crypto line shows up next year.

long AccountId instead of RedbObject<AccountProps>. redb supports object references natively, and for domain entities they're great — the whole graph loads in one call. But postings are read in the millions and always by account, so what's needed here is a fast scalar filter, not a graph. A flat long hits a partial index on numeric values and behaves like an ordinary column. The rule: use a graph where the object is read whole; use a flat key where you filter by it.

RedbListItem for currency and operation kind. These are redb's built-in lookup lists: the value is stored as a reference to a list item, and you can filter both by item identity and by its string value. No enum tables and no joins.

Decision #2: the account has no balance field

[RedbScheme(Name = "payments.account", Alias = "Account")]
public class AccountProps
{
    public string Number { get; set; } = "";
    public long OwnerId { get; set; }
    public RedbListItem? Currency { get; set; }
    public RedbListItem? AccountType { get; set; }
    public DateTimeOffset OpenedAt { get; set; }
    public DateTimeOffset? ClosedAt { get; set; }

    // There is NO Balance field here. On purpose.
}

A balance field on the account is the most common and most expensive mistake in payment systems. It immediately creates two sources of truth: the balance in the field, and the balance as the sum of postings. They diverge. Always. The only question is when you notice.

A balance is a function of the journal, full stop:

var balance = await redb.Query<PostingProps>()
    .Where(p => p.AccountId == accountId)
    .SumAsync(p => p.Delta);

One query, aggregated database-side, nothing pulled into memory.

Decision #3: the snapshot is an optimisation, not the truth

On an account with a million postings you obviously can't sum everything every time. Hence snapshots:

[RedbScheme(Name = "payments.balance_snapshot", Alias = "Balance snapshot")]
public class BalanceSnapshotProps
{
    public long AccountId { get; set; }
    public decimal Amount { get; set; }

    /// <summary>This snapshot accounts for every posting with id ≤ this value.</summary>
    public long UpToPostingId { get; set; }

    public DateTimeOffset TakenAt { get; set; }
}

The balance is then "latest snapshot plus deltas since":

var snap = await redb.Query<BalanceSnapshotProps>()
    .Where(s => s.AccountId == accountId)
    .OrderByDescending(s => s.UpToPostingId)
    .FirstOrDefaultAsync();

var baseAmount = snap?.Props.Amount ?? 0m;
var fromId     = snap?.Props.UpToPostingId ?? 0L;

var delta = await redb.Query<PostingProps>()
    .Where(p => p.AccountId == accountId)
    .WhereRedb(o => o.Id > fromId)
    .SumAsync(p => p.Delta);

var balance = baseAmount + delta;

The key property: a snapshot can be thrown away and recomputed at any moment, because the source of truth is the journal. Snapshot corrupted, stale, or computed by old logic? Delete it and take a new one. With a balance field on the account you don't get that luxury — once it diverges, you no longer know which number is right.

Snapshots are taken by a scheduled background job — an ordinary route, covered in the integrations section.

Decision #4: idempotency belongs in the model, not in a side table

The operation key lives inside the posting, not in a separate "processed messages" table. Which makes "has this operation already been posted?" an ordinary query:

var alreadyPosted = await redb.Query<PostingProps>()
    .Where(p => p.OperationId == operationId)
    .AnyAsync();

Why this beats a side table: it cannot get out of sync. The posting and the record that it happened are the same object, written by the same operation. The classic two-table shape permits "marker exists, posting doesn't" and vice versa — and that gets cleaned up by hand.

A route-level idempotent consumer still exists, but it solves a different problem: rejecting redelivery before we ever reach the database. Two different lines of defence; neither replaces the other.

The model as a whole

   ┌────────────────┐         ┌─────────────────────┐
   │    Account     │         │  BalanceSnapshot    │
   │  payments.     │◄────────┤  payments.          │
   │  account       │  1:N    │  balance_snapshot   │
   │                │         │                     │
   │ Number         │         │ AccountId           │
   │ OwnerId        │         │ Amount              │
   │ Currency       │         │ UpToPostingId  ─────┼──┐
   │ AccountType    │         │ TakenAt             │  │
   │ (no balance!)  │         └─────────────────────┘  │
   └───────┬────────┘                                  │
           │ 1:N                                       │
           │                                           │
   ┌───────▼───────────────────────────────┐           │
   │            Posting                    │           │
   │        payments.posting               │◄──────────┘
   │                                       │  snapshot "up to"
   │  OperationId  ← idempotency           │
   │  AccountId                            │
   │  Delta        ← NUMERIC(38,18)        │
   │  Currency     ← RedbListItem          │
   │  Kind         ← RedbListItem          │
   │  OccurredAt                           │
   │  ReversesPostingId ──┐                │
   │  Reference           │ reversal       │
   └──────────────────────┴────────────────┘
              APPEND-ONLY. Never updated.

Notice what the model doesn't have: an idempotency table, a change-history table, a status field on the posting, a balance field on the account. Each is absent for its own reason, and every reason is the same underlying one — don't create a second source of truth.

Transaction boundaries: where exactly the line runs

The previous section was about what to store. This one is about what has to happen atomically. This is where people get it wrong most often, and where being wrong costs the most.

There's one rule and it's rigid:

  INSIDE ONE TRANSACTION                  OUTSIDE — NEVER
  ──────────────────────                  ───────────────
  ✓ every posting of one operation        ✗ HTTP to the acquirer
  ✓ the outbox row                        ✗ publishing to a broker
  ✓ the balance snapshot update           ✗ sending an email
  ✓ the idempotency marker                ✗ any network call
    (which is the posting itself)         ✗ waiting on anyone else

The reason is grade-school simple: a transaction holds locks, and a network call can take thirty seconds. An external call inside a transaction is guaranteed degradation under load plus entertaining deadlocks at 3 a.m.

Which is also why the outbox isn't a luxury but a necessity: you cannot publish to a broker atomically with a posting, but you can write to your own database.

What it looks like in code

public async Task<TransferResult> TransferAsync(
    long fromAccountId, long toAccountId, decimal amount,
    string operationId, CancellationToken ct)
{
    // 1. Cheap rejection BEFORE the transaction: shed retries without taking locks
    if (await redb.Query<PostingProps>()
            .Where(p => p.OperationId == operationId).AnyAsync())
        return TransferResult.AlreadyProcessed;

    await using var tx = await redb.Context.BeginTransactionAsync();

    // 2. Lock accounts STRICTLY IN ASCENDING id ORDER.
    //    Otherwise two opposing transfers A→B and B→A deadlock.
    var locked = new[] { fromAccountId, toAccountId };
    Array.Sort(locked);
    await redb.LockForUpdateAsync(locked);

    // 3. Re-check — now under the lock (TOCTOU guard)
    if (await redb.Query<PostingProps>()
            .Where(p => p.OperationId == operationId).AnyAsync())
        return TransferResult.AlreadyProcessed;   // dispose rolls back

    // 4. Sufficient funds — evaluated under that same lock
    if (await GetBalanceAsync(fromAccountId) < amount)
        return TransferResult.InsufficientFunds;

    var now = DateTimeOffset.UtcNow;

    // 5. Both postings in one batch, one round-trip
    await redb.AddNewObjectsAsync(new[]
    {
        Posting(operationId, fromAccountId, -amount, now),
        Posting(operationId, toAccountId,   +amount, now),
    });

    // 6. The event goes to the outbox IN THE SAME TRANSACTION
    await redb.SaveAsync(OutboxEvent("transfer.completed", operationId, now));

    await tx.CommitAsync();
    return TransferResult.Posted;
}

The non-obvious parts.

The double idempotency check isn't paranoia. The first, outside the transaction, sheds mass retries cheaply: it takes no locks and blocks nobody. The second, under the lock, closes the window between check and write. Without the first the system degrades on a retry storm; without the second it lets a double posting through.

Sorting ids before locking is mandatory. Textbook, and routinely forgotten. Two opposing transfers — A→B and B→A — lock the accounts in opposite order and wedge. A single canonical acquisition order removes an entire class of deadlocks; the store itself uses the same trick internally for batch operations.

The balance check sits under the same lock as the write. Check without the lock and you get the classic race that produces a double debit.

Both postings go in one batch. AddNewObjectsAsync is a single round-trip rather than two. On a transfer the difference is small; on bulk charging it's decisive.

If you don't need explicit rollback control, there's a shorter form:

await redb.Context.ExecuteAtomicAsync(async () =>
{
    await redb.AddNewObjectsAsync(postings);
    await redb.SaveAsync(outboxEvent);
});

Why this matters for sharding later

Notice that the entire transaction fits inside one database. That isn't an accident — it's a requirement you have to design in from day one.

There is no distributed transaction across shards in cross-platform .NET; two-phase commit over two PostgreSQL instances isn't available. So if the platform ever grows sideways, the shard key must be chosen so both sides of a transfer land on the same shard. In practice that means sharding by customer or by account group, not by payment id.

You make this decision once, at the start. Redoing it on live data is a separate project.

The boundary, drawn

   HTTP request
       │
       ▼
   ┌──────────────────────────────────────────────┐
   │  1. Idempotency check (no locks taken)       │  ← outside the transaction
   └───────────────────┬──────────────────────────┘
                       │
   ╔═══════════════════▼═══════════════════════════╗
   ║                TRANSACTION                    ║
   ║                                               ║
   ║   2. LockForUpdate(accounts, ascending id)    ║
   ║   3. Idempotency check, again                 ║
   ║   4. Sufficient-funds check                   ║
   ║   5. AddNewObjects(postings)                  ║
   ║   6. Save(outbox event)                       ║
   ║                                               ║
   ║        COMMIT ────────────────────────────┐   ║
   ╚═══════════════════════════════════════════╪═══╝
                                               │
                       ┌───────────────────────┘
                       │ from here on: async, outside the transaction
                       ▼
   ┌───────────────────────────────────────────────┐
   │  Sql.Poll(outbox) → Kafka → acquirer / bank   │
   │  Wait on the network as long as you like:     │
   │  locks are released, money is already posted  │
   └───────────────────────────────────────────────┘

Module split: unit of deploy = unit of failure

A module in Tsak is a .tpkg: a built package with an entry point that the runtime loads into its process, giving it its own route context, its own assembly load context, and its own DI container.

The slicing criterion is simple, and it's neither "layers" nor "domains":

A module is whatever you'll want to deploy or roll back separately from everything else.

Which naturally produces a split by external protocol rather than by business entity.

   ┌─────────────────────────────────────────────────────────┐
   │  payments.Core                                          │
   │  ────────────                                           │
   │  • schemas: Account, Posting, BalanceSnapshot, Outbox   │
   │  • ledger: TransferAsync, GetBalanceAsync, Reverse      │
   │  • lookups: currencies, operation kinds                 │
   │  • NOT ONE external transport                           │
   │                                                         │
   │  Entrances: direct-vm://payments-post                   │
   │             direct-vm://payments-balance                │
   └────▲──────▲──────────▲──────────▲──────────▲────────────┘
        │      │          │          │          │
        │      │          │          │          │  direct-vm://
        │      │          │          │          │
   ┌────┴───┐ ┌┴──────┐ ┌─┴──────┐ ┌─┴──────┐ ┌─┴────────┐
   │  .Api  │ │ .Acq  │ │ .Bank  │ │ .Files │ │  .Recon  │
   │        │ │       │ │        │ │        │ │          │
   │ HTTP   │ │ HTTPS │ │ IBM MQ │ │ SFTP   │ │ Quartz   │
   │ intake │ │acquirer││  bank  │ │registry│ │  recon   │
   └────────┘ └───────┘ └────────┘ └────────┘ └──────────┘
    facade     outbound   outbound   files     scheduled

What that buys in practice:

The acquirer changed its format — payments.Acq gets redeployed. The bank rail, payment intake and reconciliation don't notice: their modules never restarted.

The ledger changes least often and ships most carefully. payments.Core is the only module that touches money. Its release cadence differs from the rest, and that's fine — it also gets the fewest changes.

A new acquirer is a new module, not an edit to an existing one. It cannot break the working one because it physically lives elsewhere.

A dead SFTP server won't take down payment intake. Each module has its own route context and its own connections.

The module entry point

public static class InitRoute
{
    public static IRouteContext main(IRouteContext context)
    {
        // Only the transports THIS module needs
        context.AddComponent(new HttpComponent());

        // A named store instance: payments has its own database,
        // identity has its own — they share neither connections nor caches
        var redb = context.GetRedbService("payments");

        context.AddRouteBuilder(new PaymentsApiRouteBuilder());
        return context;
    }
}

The named store isn't a detail — it carries an important property: the instance is created per exchange and lives exactly as long as the message is being handled. The connection never becomes a shared resource, and concurrent requests don't queue behind a single one.

How modules call each other

Inside one worker, without the network:

public class PaymentsApiRouteBuilder : RouteBuilder
{
    protected override void Configure()
    {
        From(Http.Listen("/api/payments/transfer").Port(8080).InOut())
            .RouteId("api-transfer")
            .Unmarshal(typeof(JsonMessageSerializer), typeof(TransferRequest))
            .Process(RequireScope("payments:write"))
            .IdempotentConsumer(e => e.Message.GetHeader<string>("Idempotency-Key"))
            .To("direct-vm://payments-post")          // ← into the ledger, no network
            .Marshal(typeof(JsonMessageSerializer))
            .Respond();
    }
}

direct-vm:// is an in-process transport between route contexts. No socket, no serialization, no TLS: same thread, same exchange object.

And here's the architectural payoff: if tomorrow payments.Core has to move to its own worker, a URI string changes, not code. direct-vm://payments-post becomes rabbitmq://payments-post or grpc://.../Post and that's it. The decision of "monolith or distributed" is deferred to operations and changed by configuration.

That's arguably the whole reason to slice modules this way.

Integration routes

The outbox — and why it isn't a redb object

The one place we deliberately step away from the typed store into a flat table:

CREATE TABLE payments_outbox (
    id           bigserial PRIMARY KEY,
    event_type   text        NOT NULL,
    operation_id text        NOT NULL,
    payload      jsonb       NOT NULL,
    created_at   timestamptz NOT NULL DEFAULT now(),
    processed    boolean     NOT NULL DEFAULT false,
    processed_at timestamptz
);
CREATE INDEX ix_outbox_pending ON payments_outbox (id) WHERE processed = false;

Why, when the whole domain lives in redb: the outbox is infrastructure, not domain. It's flat by nature, lives for seconds, gets read in batches by one predicate, and gets deleted. It has no use for typing, object graphs or schema evolution — and a lot of use for a partial index on unprocessed rows and batched polling.

This is exactly the situation where it matters that the storage structure is open: you can write to redb and to plain SQL in the same transaction, because the context and the connection are shared.

await redb.Context.ExecuteAtomicAsync(async () =>
{
    await redb.AddNewObjectsAsync(postings);          // domain → redb

    await redb.Context.ExecuteAsync(                  // infrastructure → SQL
        "INSERT INTO payments_outbox (event_type, operation_id, payload) VALUES (@t, @o, @p)",
        eventType, operationId, payloadJson);
});

The rule that falls out: redb for what you read, search and evolve. Flat SQL for what you pump through. Mixing them in one transaction is fine and normal.

Then the publisher:

From(Sql.Poll("SELECT * FROM payments_outbox WHERE processed = false ORDER BY id LIMIT 200")
        .DataSource("payments")
        .Delay(500)
        .OnSuccess("UPDATE payments_outbox SET processed = true, processed_at = now() " +
                   "WHERE id = ANY(@ids)")
        .Transacted())
    .RouteId("outbox-publisher")
    .Split(Body())
    .To(Kafka.Topic("payments.events")
        .Acks("All")
        .EnableIdempotence(true)
        .EnableTransactionalProducer(true)
        .TransactionIdPrefix("payments-outbox"));

OnSuccess runs in the same transaction scope as the publish: either the event went out and is marked processed, or neither. There's no in-between state.

Outbound to an acquirer: where the save-point goes

From(Kafka.Topic("payments.events").GroupId("acq"))
    .RouteId("acq-charge")
    .Filter(Header("event_type").isEqualTo("transfer.completed"))

    .Replayable("acq-charge")            // ← SAVE-POINT
        .Process(BuildAcquirerRequest)
        .Retry(3).RedeliveryDelay(2000).UseExponentialBackOff()
        .To(Http.Post("https://acq.example.com/v1/charge").Timeout(15_000))
        .Process(ParseAcquirerResponse)
        .To("direct-vm://payments-mark-charged")
    .EndReplayable();

The save-point goes after entering the route and before the first external call — that is, at the boundary where state is assembled but we haven't gone outside yet. If the acquirer is down, retries don't help and the exchange fails, the snapshot lands in the dead-letter store and a support operator hits Replay from the dashboard once the acquirer is back.

An important detail the engine checks for you: don't casually put a save-point on a route where a broker or a transaction already owns redelivery. Otherwise two parties are responsible for the retry, which is the direct path to a double charge. This route deliberately isn't marked .Transacted() for exactly that reason: the save-point mechanism owns the retry here, not Kafka.

The bank on IBM MQ: here it's the opposite

From(Wmq.Queue("PAYMENTS.IN").QueueManager("QM.PROD").Transacted(true))
    .RouteId("bank-inbound")
    .Transacted()                          // ← ack and send commit together
    .Unmarshal(typeof(Iso20022Codec))
    .ValidateXsd(Schemas.Pacs008)          // schema validation is a built-in step
    .Process(MapToPostingRequest)
    .To("direct-vm://payments-post")
    .To(Wmq.Queue("PAYMENTS.ACK").Transacted(true));

The model here is the inverse of the previous one: the queue owns redelivery. On failure everything rolls back, the message returns to the queue, the backout counter increments, and past a threshold the message moves to a dedicated queue for triage instead of looping forever.

You write the ISO 20022 parsing (no codecs ship in the box — every profile differs), but XSD validation is built in, and that's half the work.

Registry files over SFTP

From(Sftp.Poll("/in/registry").Include("*.xml").Delay(60_000).Move("/in/done"))
    .RouteId("files-registry")
    .ValidateXsd(Schemas.Registry)
    .Split(XPath("//Payment"))
        .Threads(8)                        // fan the parsing out
        .Process(MapToPostingRequest)
        .To("direct-vm://payments-post")
    .EndSplit()
    .To("log://registry-done");

.Threads(8) matters here: the source polls serially, but parsing a ten-thousand-line registry fans out across a pool. Ordering within the registry is lost — acceptable for charges, not acceptable for a sequence of operations on one account. If you need that, parallelise by account, not by line.

Scheduled reconciliation

From(Quartz.Cron("0 30 3 * * ?"))          // daily at 03:30
    .RouteId("recon-daily")
    .Process(async (e, ct) =>
    {
        var redb = e.Context.GetRedbService("payments", e);

        var byMerchant = await redb.Query<PostingProps>()
            .WhereRedb(o => o.DateCreate >= DateTime.Today.AddDays(-1))
            .GroupBy(p => p.MerchantId)
            .SelectAsync(g => new { g.Key, Total = Agg.Sum(g, p => p.Delta) });

        e.Message.SetBody(byMerchant);
    })
    .To("direct-vm://recon-compare")        // compare against the acquirer statement
    .To(Sql.Insert("recon_report"));

The aggregation runs database-side over live data. No separate analytical stack is needed for this — you'll want one later, for something else, which we'll get to.

The route map

  IN                      LEDGER                   OUT
  ──                      ──────                   ───

  HTTP /transfer ──┐
                   │
  IBM MQ PAY.IN ───┼──► direct-vm://          ┌──► payments_outbox
   (transacted)    │    payments-post ────────┤    (same transaction)
                   │         │                └──► postings in redb
  SFTP registries ─┘         │
   (Threads 8)               ▼
                        ┌─────────┐
  Quartz 03:30 ────────►│  redb   │
   (reconciliation)     │payments │
                        └─────────┘
                             ▲
                             │
       ┌─────────────────────┘
       │  Sql.Poll(outbox) ──► Kafka (EOS) ──┬──► acquirer (HTTP)
       │   .Transacted()                     │     └ .Replayable
       │                                     └──► notifications
       └─ OnSuccess: processed = true

The facade: where authorization sits

The identity server runs in the same worker but on its own database. That matters: payment data and OAuth records share no connections, no transactions, no caches. Splitting them onto separate database servers later is a connection-string change, not a rewrite.

   EXTERNAL CLIENTS                   INTERNAL MODULES
   ────────────────                   ────────────────
   browser, mobile app,               payments.Api
   partner                            payments.Acq
        │                                   │
        │ HTTPS                             │ direct-vm://
        │ (the spec requires                │ (no network,
        │  a browser only for               │  no TLS,
        │  /authorize)                      │  no JSON)
        ▼                                   ▼
   ┌──────────────────────────────────────────────────┐
   │              redb.Identity                       │
   │                                                  │
   │  /connect/token      direct-vm://identity-token  │
   │  /connect/introspect direct-vm://identity-...    │
   │  /connect/authorize  ← the only one needing HTTP │
   │  /scim/v2/*                                      │
   │                                                  │
   │  79 typed audit events ──► redb + SIEM           │
   └──────────────────┬───────────────────────────────┘
                      │
              ┌───────▼────────┐
              │  identity DB   │   ← separate from payments
              └────────────────┘

Authorization at the payment API entrance:

.Process(async (e, ct) =>
{
    var token = e.Message.GetHeader<string>("Authorization")?["Bearer ".Length..];

    // Introspection WITHOUT a network call — same process
    var result = await _identity.RequestBody<IntrospectionResponse>(
        IdentityEndpoints.Introspect, new { token });

    if (result?.Active != true || !result.Scopes.Contains("payments:write"))
        throw new UnauthorizedException();

    e.SetProperty("subject", result.Subject);
})

The difference from the usual shape isn't cosmetic. Classically every internal call means a network hop to the identity server — and at meaningful traffic that becomes a bottleneck, a single point of failure, and a permanent latency tax. Here it's a method call.

From the outside it stays a normal OIDC provider: the browser parts speak HTTPS because the spec says so, and everything else — issuance, refresh, introspection, revocation, management, SCIM — is transport-neutral.

Which also implies something worth keeping in mind when designing closed segments: you can have as many front doors as you have channels. An inter-branch segment with regulator-mandated crypto, a site where only the corporate bus is allowed, a partner channel with a proprietary format — those are separate adapter modules in front of one shared core. One core, one set of permissions, one audit trail; only the adapters differ.

Cluster topology: nodes don't have to be identical

Here operations begins, and here's a property that rarely gets designed in — unfairly so.

Tsak's coordinator is three-level: cluster → group → node. A group is a geographic or logical partition. And placement is tracked per module, scoped to a group — so a node isn't a replica of its neighbour, it's a carrier of a specific module set.

   cluster: production
   │
   ├── group: acquiring          ← hot path, many nodes
   │    ├── node-acq-1   [payments.Api, payments.Acq, payments.Core]
   │    ├── node-acq-2   [payments.Api, payments.Acq, payments.Core]
   │    └── node-acq-3   [payments.Api, payments.Acq, payments.Core]
   │
   ├── group: payouts            ← bank rail, separate network
   │    ├── node-pay-1   [payments.Bank, payments.Files, payments.Core]
   │    └── node-pay-2   [payments.Bank, payments.Files, payments.Core]
   │
   └── group: reporting          ← scheduled work, one node is enough
        └── node-rep-1   [payments.Recon, identity]

   Leader: elected cluster-wide, with epoch fencing.
           A stale leader cannot corrupt state after
           losing the election.

What this gives architecturally:

Contours are separated by load and criticality without being split into separate systems. The acquiring contour scales horizontally for peaks, the bank rail lives on two nodes in a separate network, scheduled work sits on one. Still one cluster, one control plane, one audit trail.

The ledger module appears in two groups. Both intake and payouts need it. That's fine — the module is stateless, state lives in the database.

The scheduler is cluster-aware. Scheduled reconciliation is marked as a leader-only job and doesn't fire on three nodes at once.

The configuration is the same on every node; node id, address and group differ:

{
  "ConnectionStrings": { "Postgres": "Host=db.cluster;Database=payments;..." },
  "Tsak": {
    "Storage": { "Type": "Redb" },
    "Cluster": {
      "Enabled": true,
      "ClusterName": "production",
      "GroupName": "acquiring",              // ← differs per group
      "NodeId": "node-acq-1",                // ← differs per node
      "ApiEndpoint": "http://node-acq-1:9090",
      "HeartbeatIntervalSeconds": 15,
      "DeadNodeTimeoutSeconds": 60,
      "LeaderLockTtlSeconds": 30
    },
    "HotReload": { "RollingUpdate": true },  // roll node by node
    "Auth": { "Enabled": true }
  }
}

Taking a node out for maintenance is draining, not switching off:

tsak cluster cordon node-acq-2     # takes no new work, finishes what it started
# ... maintenance ...
tsak cluster uncordon node-acq-2

Failure map: what breaks, and what handles it

An architecture is judged not by how it works but by how it fails. Walking the scenarios:

What happened	What happens	Who fixes it
Process died between posting and publish	Posting is committed, outbox row isn't marked. The publisher picks it up on start	Itself
Duplicate message from the broker	Idempotent consumer rejects by key; if it slips through, the `OperationId` check under the lock catches it	Itself, two lines
Acquirer returns 500	Three retries with exponential backoff. Still failing → snapshot to the dead-letter store	Support, one click
Acquirer down for an hour	Same, but snapshots pile up. Once it's back — bulk replay	Support
Bank returned an error over MQ	Transaction rolls back, message returns to the queue, backout counter climbs; past the threshold it moves to a triage queue	Itself + triage
Two opposing transfers at once	Accounts locked in one canonical ascending order — no deadlock	Itself
Node died	Heartbeats stopped, the leader reassigned its modules to live nodes	Itself
The leader died	Re-election once the lock TTL expires; the epoch increments, so the old leader can't do damage	Itself
Balance snapshot diverged	Delete and recompute: the journal is the source of truth	Scheduled job
Shipped a bad module version	Upload the previous `.tpkg`; the old context drains, the new one starts	One command
Unsigned module in the directory	Rejected at the load boundary, before a single line of its code runs	Itself

Look at the distribution in the right-hand column: the overwhelming majority is handled by mechanisms rather than by a person. A human is needed where the failure is someone else's — and that isn't ours to fix.

Where save-points go, and where they don't

A rule worth pinning in your review checklist:

   .Transacted()        ─── broker/transaction owns the retry
                            → do NOT add a save-point
                            (two parties owning retry = double charge)

   .Replayable("name")  ─── we own the retry
                            → place it AFTER the entrance,
                              BEFORE the first external call

   neither              ─── no retry at all
                            → a deliberate decision, not an oversight

The engine will warn if a save-point ends up on a transactional route — but it's better caught at review.

The decision checklist

Everything you'll have to decide, in the order to decide it. The first three are the most expensive to change later.

1. The shard key — before you need sharding. Both sides of a transfer must land on the same shard: there will be no distributed transaction. Shard by customer or account group, not by payment id.

2. Postings are append-only. No statuses, no edits. Mistakes are corrected by reversal. This also disposes of the change-history question — the journal is the history.

3. There is no balance field. Journal plus snapshot-as-cache only. The snapshot must be disposable and recomputable.

4. Idempotency on two lines. At the route, against redelivery. In the model, as the operation key inside the posting, checked under the lock.

5. One canonical lock acquisition order. Ascending id, always.

6. No network calls inside a transaction. Everything external goes behind the outbox.

7. Flat outbox, typed domain. Mixing them in one transaction is fine.

8. Slice modules by external protocol. Unit of deploy = unit of failure.

9. The ledger core has no transports. Every entrance is in-process — then extracting it into its own worker becomes a URI change.

10. The identity server gets its own database. Splitting later is a connection string.

11. Cluster groups by contour, not by sameness. Nodes carry different module sets.

12. Save-points only where we own the retry.

What's out of scope

Honest boundaries of this reference model.

Your accounting model will differ. The one shown is minimal: accounts, postings, snapshots. A real one grows a chart of accounts, analytical dimensions, multi-currency with revaluation, fund reservation and pricing rules. That's your domain, and no vendor will design it for you.

You write the financial format codecs. Transport, framing and schema validation exist; parsing a specific ISO 20022 or ISO 8583 profile is yours. It would have been yours anyway — every profile differs.

Regulated open banking needs work. Mutual-TLS client authentication, rich authorization requests, decoupled confirmation, and step-up authentication for a transaction aren't implemented. The cryptographic groundwork is in the server; this is a scoped sprint.

An analytical stack will appear eventually. Built-in aggregation covers operational queries — remaining limit, today's gap, a window for a fraud rule. A yearly slice across ten dimensions doesn't get computed on a live operational database on any storage engine, and you will build data marts. Just not in year one and not as a launch precondition — and "roll up aggregates and write them flat" is an ordinary scheduled route.

Sharding is a kit. The mechanisms exist: block key generation with the ability to move the key source into a separate database, per-connection cache isolation, several independent stores in one module. An assembled solution doesn't. That's a post of its own, in progress — including why the key authority can be an empty database with one line of DDL, and why a failure there is fixed with one SQL statement.

Wrapping up

The reference model in one paragraph: append-only postings with the balance as a function of them; a transaction containing not one network call; the outbox as the bridge into the asynchronous world; modules sliced by external protocol; a ledger core with no transports, reached in-process; a cluster whose nodes carry different module sets; and save-points exactly where we own the retry.

None of this is invention — it's discipline familiar to anyone who has built payment systems. The difference is how much code it takes to implement: transactions, locks, exact arithmetic, idempotency, an outbox, transports, save-points, clustering and an operations console are already here, and what's left is describing your ledger.

For what that layer costs if you write it yourself, see the companion post with the full breakdown.

If you're building something similar — tell me in the comments which decisions you made differently and why. The forks where you chose otherwise and didn't regret it are more useful than any feature list.

Sources and releases: github.com/redbase-app. About the store: redbase.app. Previous posts: my dev.to profile.

If this was useful — a ⭐ on GitHub helps others find it.

A payment platform on .NET: what it actually costs, and how much of it you never have to write

rinat kozin — Tue, 28 Jul 2026 19:30:24 +0000

Conversations about payment platforms almost always start from the wrong end. People argue message brokers. They debate microservices versus a modular monolith. They draw the schema.

Eighteen months later a team of twelve has produced a million and a half lines, of which maybe two hundred thousand are business logic. The rest is transport, retries, dashboards, deployment, and increasingly elaborate attempts to work out where a payment got stuck last night.

This post is about that second number. About the layer that earns nothing and without which nothing runs — and about how much of it you can simply not write in 2026.

We've spent four years building redb: a typed store on top of PostgreSQL, MS SQL and SQLite; an integration engine; a runtime with clustering and a dashboard; and an identity server speaking OAuth 2.1 and OpenID Connect. It runs in our own production, ships as packages and images, and — relevant to everything below — every Pro capability is free across the whole 3.x line, no keys and no license server.

What follows is a payment platform taken apart by layer, with an honest accounting of where the person-years go. No code: the technical deep dives live in other posts. This one is about money, timelines and risk.

Part of the redb / redb.Route series — recent posts first:

redb 3.4.0: day-two operations for a .NET stack

redb 3.3.0: an enterprise .NET stack you actually own

Leaving MassTransit for a Camel state of mind: the Kafka connector, Scatter-Gather, and transactions

Apache Camel for .NET, dissected: the HTTP connector with no ASP.NET MVC + the Content-Based Router

redb.Route — Apache Camel for .NET: 22 transports, 30+ EIP patterns, compiled DSL

Sources: github.com/redbase-app. Docs: redbase.app.

Three layers, three budgets

Every payment platform — a marketplace's payouts, a wallet, a PSP, a corporate treasury, a SaaS billing engine — decomposes into three parts, and each has a different cost profile.

The core. Holds money and operations. Postings, balances, limits, pricing, reconciliation. This is what you get paid for, and it's where a mistake costs the most: a mis-rounded cent isn't a bug, it's a complaint.

The facade. Decides who gets in. Who is this, what are they allowed to do, how did they prove it, and how is all of that recorded for the auditor and the security team. A layer made entirely of standards, invisible to the business — right up until the first audit.

The integrations. Talk to the outside world. A bank on IBM MQ, an acquirer over HTTP, payout files on SFTP, accounting on a schedule, fraud scoring on a queue. The layer that breaks most often and costs the most when it does, because the thing on the other end belongs to someone else.

Layer by layer: what genuinely has to be yours, and what's already written.

Layer one: the core, where cents don't evaporate

Start with something that looks like an implementation detail and is actually a first-order business risk: how the number that measures money is defined.

In most .NET projects an amount lands in the database as a decimal with whatever precision someone set in the ORM config years ago. Usually two decimal places, sometimes four. Then: a fee is a percentage, the percentage produces a third decimal, the third decimal gets rounded, roundings accumulate — and a year later reconciliation with the acquirer is off by an amount nobody can explain. That's not a hypothetical. It's a genre, and it gets resolved by hand, by an analyst and a developer, over weeks.

In redb, monetary precision isn't in application config. It's in the storage schema itself: numeric values live in a column with 38 digits of precision, 18 of them after the decimal point. The schema comment on that column says why, in as many words: exact decimal numbers for financial calculations, lossless, for money, taxes and percentages where arithmetic accuracy matters.

In business terms:

Fees, FX and tax compute without accumulating rounding error. Eighteen decimals is headroom for any chain of calculations, not "two decimals and fingers crossed."
Multi-currency and crypto work out of the box. Eighteen decimals is exactly the precision Ether is denominated in. If a crypto line of business shows up next quarter, the storage doesn't change.
Precision isn't a function of who configured the project. A developer can't accidentally declare an amount with two decimals, because the decision was made one level below them.

Money doesn't get double-spent

The second baseline guarantee: one operation executes once, and never half-way. This part is standard and boring, as finance should be — real transactions with explicit control, atomic execution of a group of operations, and row-level locking while a record is being modified.

That last one is worth calling out, because it's where homegrown solutions usually break. Classic scenario: two requests read the balance concurrently, both see "sufficient funds", both debit. The defence is locking the row for the duration. In redb that's a first-class mechanism, and it isn't decorative — it's what our own subsystems run on: the identity server's token store, the failed-MFA-attempt counter, the cluster's distributed lock. It carries production load for us, rather than existing "in case someone needs it."

Deletion you can undo

Another thing that gets reinvented in every finance project: correct deletion. You can't hard-delete — reporting, regulators, incident forensics. So it's either an is_deleted flag dragged into every query forever, or archive tables and the job of keeping them in sync.

In redb this is built in, in two phases. First the object is marked — together with the whole tree of records hanging off it — and immediately disappears from every normal query, with no change to a single select in your code. Physical removal happens later, on a background worker, in batches, with progress persisted in the database, so a restart doesn't kill it and any node in the cluster can pick it up.

Two business effects: the UI can honestly offer "Deleted — undo" and have it work, and the user-facing delete doesn't start crawling as data grows.

Reporting where the data already is

Reconciliation, registries, regulatory reports, commission calculations — all aggregation. The usual answer is a separate stack: extract, warehouse, BI. A separate stack means a separate team, separate licenses, and the eternal question of why the report disagrees with the system.

redb computes aggregates, groupings, window functions and having-clauses on the database side, over the same data serving operations. "Show me merchants whose reconciliation gap exceeds the threshold" is a query against live data, not an export into a third system.

The fair objection — and what to do with it

Here an experienced reader will reasonably point out that aggregating over an operational store, at volume, is slower than aggregating over a flat pre-built table. That's true, and there's nothing to argue about.

Except the objection is about a different problem. Nobody does heavy analytics on an operational core, and not because of the storage engine — because it's bad architecture on any storage engine. A yearly slice across ten dimensions doesn't get computed on the live database on plain PostgreSQL, or Oracle, or anything else: it competes for resources with payment processing, and payments win. For that, you build flat data marts. You still will.

The two query classes get conflated in these arguments, so let's separate them:

Operational. Narrow slice, fresh data, answer needed now: remaining limit, today's reconciliation gap, top declines in the last hour, windowed sum for a fraud rule, ranking operations inside a batch. Small volumes of hot data — and these need to compute where the data lives, or you end up with a data mart that's a day stale answering a question that needed a second.

Analytical. Wide period, many dimensions, tolerates latency: reporting, product analytics, finding patterns. That's a data mart, and it should be.

Built-in aggregation is aimed squarely at the first class. It covers the case where a typical project either stands up a warehouse prematurely, hand-writes optimized SQL, or pulls rows into the app and sums them in memory.

And the mart itself isn't a project. "Every hour, roll up aggregates and write them flat into a table or a search index" is an ordinary route on this stack: a scheduler, a read, a sink. Configuration, not an ETL programme with its own headcount.

So data marts stay exactly where they belong. They just stop being a precondition for launch — which takes them out of year-one budget and puts them in the year they're actually justified.

Nothing is hidden: there's always a way down

A separate guarantee worth stating, because it removes a whole class of risk.

Any abstraction over a database eventually meets a query it can't express. Then you get two options: a workaround in application code, or a ticket to the vendor with an unknown ETA. In a financial system that moment arrives on schedule — at the first non-standard request from security, compliance, or a regulator.

There's no dead end here. The storage structure is open and documented, and you can always go at it with plain SQL — your analyst, your DBA, your existing tooling.

And that isn't "technically possible but painful." The structure is designed so odd queries come out short and fast.

The showcase example is a query that's nearly impossible in a classical schema: find every object that has any field starting with a given string, regardless of what type those objects are. In a normal relational model that's a union across forty tables enumerating every text column — or a separate full-text system you now have to feed and keep in sync. Here, values for all objects of all types live in one table with a pattern-search index over it. The answer comes back immediately, across millions of rows, without caring which entity anything belongs to. Same story for "which objects have this field populated at all" — there are dedicated indexes for exactly that check.

What this buys:

An unplanned request stops being a sprint. Compliance wants operations matching an odd criterion, security is reconstructing an incident, a regulator asked for something in their own phrasing — that's an analyst-hour, not a backlog item for next quarter.
Custom logic is cheap. Views, stored procedures, feeds into your reporting — with ordinary database tooling, without asking us and without waiting for a release.
The data isn't locked in. The database reads with standard PostgreSQL or MS SQL tools, without a single line of our code.

The storage layout is written up in full at redbase.app/architecture.

What you write yourself anyway — and why that's correct

There's no double-entry, no chart of accounts, no posting rules in the box. That's not a gap, it's a boundary.

The accounting model is your product. A marketplace's is one thing, a treasury's another, a card issuer's a third. Every attempt by a vendor to hand you a "ready-made ledger" ends with the business spending six months bending its model to fit someone else's abstractions, then a year living with the workarounds where it didn't fit.

What redb provides is the right foundation underneath your model: an exact number, transactions, locks, reversible deletion, and aggregation. You describe postings as ordinary C# classes, and they become tables.

Which brings us to the second effect, and the one businesses consistently under-weight.

Changing the schema stops being an event

In a normal project, adding one field to a payment entity looks like this: a developer writes a migration, the migration goes to review, review pulls in a DBA, the DBA asks for a rollback script, the script gets tested on staging, the deploy waits for the next window. Between "we need a field" and "the field is in production": days to weeks, in a healthy organisation. In a bank, longer.

In redb the schema is described by ordinary C# classes, and the store reconciles itself against them. Add a property, it exists. No migration, no DDL, no sign-off.

Three consequences, all of them money.

One: designing as you go. The classical approach demands an agreed data model before development starts. In a financial platform that model isn't knowable up front — it emerges as you discover what the partner bank actually sends and what the business actually needs on screen. So you either spend a month or two guessing, or you get a stream of migrations later. Usually both.

When the schema can evolve freely, that phase never happens. The team starts with a base set of fields and grows it as understanding arrives. On a payment-platform scale that's 400–900 engineering hours — before a single line of useful code.

Two: being wrong about the model stops being expensive. The classic: a field was designed as text, and a month later it needs to be a reference. In a normal project that's a migration with data conversion, a backfill and a rollback plan — a day to several, plus production risk. So in practice that decision often doesn't get made: the answer is "too expensive to change", and the workaround lives forever.

Here, changing a field's type is editing a class. And the Pro tier ships a mechanism that backfills a new field from data you already have: add "amount including tax" and it populates from amount and rate across the entire operation history, expressed in C#, with no export and no script.

Three, and most under-rated: the cost of testing an idea.

Work out what it costs to test a product hypothesis that needs a new field or a new entity. Think of it, write the migration, spin up a local stack or fight for the shared one, run it, and if it didn't work — write the reverse migration and clean up the data. Honestly: half a day to a day.

Here: add a property, run against a local database file, didn't work, delete the property. Twenty minutes.

An order of magnitude. And the point isn't the saved hours, though across a project's life that's another 500-plus. The point is how many options the team gets to consider at all. At half a day per test, you evaluate one and ship the first thing that works. At twenty minutes, you evaluate three to five and ship the best one. That's not a line item, that's product quality — and it resurfaces later in conversion, in support volume, and in what rework costs a year out.

On the local stack specifically: the same store runs on PostgreSQL, MS SQL and SQLite with no code changes. So a developer fixing one small module doesn't need a full environment of database, broker, identity server and neighbouring services — a file will do. Onboarding drops from days to hours, and CI stops needing infrastructure spun up per pull request.

Layer two: the facade the auditor will look at

Nobody sane writes their own authorization server anymore. You take one off the shelf — and that's where a fork in the road costs money before the first line of code.

The commercial .NET product means a license, and for a financial institution that license isn't on the entry tier. Open-source Keycloak means a Java stack inside a .NET organisation: another runtime, another skill set, another on-call rotation, another upgrade cycle. Both work. Both cost.

redb.Identity is OAuth 2.1 and OpenID Connect inside the same stack, on the same storage, under the same runtime. What matters to the business here:

Standards in the code, not on a slide. The source references 40 RFCs by number — not as a wish list, but next to the implementation and the tests that check it. The full set a security reviewer asks about: mandatory PKCE, proof-of-possession token binding, pushed authorization requests, dynamic client registration, private-key-JWT client authentication, token exchange, the device grant, introspection, revocation, back-channel logout.

Verified by an outside arbiter. The server is exercised against the official OpenID Foundation conformance suite — the same suite the Foundation uses to certify providers. The configuration profile passes with zero findings; the authorization-code modules of the basic profile are green. Formal certification is the next step and we're not claiming it as done. But the fact of running against an external reference is something most .NET stacks, homegrown and purchased alike, never do — and on an audit that's a different quality of conversation.

Audit and second factors included. MFA: TOTP, SMS and email codes, FIDO2 hardware keys, recovery codes. User lifecycle over SCIM, so HR systems provision and deprovision automatically. And 79 typed audit event types across seven categories, persisted and optionally fanned out to your SIEM in parallel. Not "logs you could reconstruct something from" — typed events: login failed, second factor failed, token replay detected, client secret rotated, all sessions revoked.

One product on three databases. The server's full test suite — over 1,700 checks — passes identically on PostgreSQL, MS SQL and SQLite by flipping one setting. If picking a database is its own conversation with infrastructure at your company, that removes a class of blockers.

The saving that never shows up in a budget

There's an architectural detail that looks technical and converts directly into hardware.

In the usual shape, every internal call between services is authorized, and authorizing means a network hop to the identity server. At meaningful traffic that server becomes a bottleneck, a single point of failure, and a permanent addition to your latency budget.

Here, modules running in the same runtime reach the identity server in-process — no network at all. No socket, no TLS handshake, no serializing to yourself. Externally it stays a normal server on standard protocols: the browser parts speak HTTP because the spec requires a browser, and everything else — token issuance, refresh, introspection, revocation, management, directory — is transport-neutral.

Our own measurements: a three-node configuration sustains roughly five thousand token requests per second at 50–80 ms p99. Translated: fewer nodes doing authorization, and one less single point of failure.

You build the front door — on your transport, with your crypto

This is an architectural fork whose consequences reach well past finance, into any distributed system whose nodes talk over more than one kind of channel.

In a typical identity server, protocol and transport are welded together: it speaks HTTPS, and that's the extent of it. Here transport is decoupled from protocol — HTTP is just the first adapter, not part of the server. Everything except the flows the spec requires a browser for is transport-neutral.

One source of truth about permissions, many front doors. A real distributed system rarely talks over one channel. Some nodes sit in a closed segment behind a tunnel. Some sites are wired to a corporate message bus because that was decided a decade ago and won't be revisited. Partner integrations go over the public internet. Edge devices don't speak modern HTTP at all.

A conventional identity server assumes one channel and that it's HTTPS. Which leaves two bad options: force everything onto HTTPS — and open up the closed segment — or run several servers for several channels, and now you have several sources of truth about who may do what. The second is worse: permission drift between segments gets discovered after the fact, and unpleasantly.

Here the core is single — the standard one, the one that passes conformance and reads sensibly to an auditor. And it has as many front doors as you have channels, each built for its channel: your transport, your envelope, your encryption, your framing. Unwrap on the way in, wrap on the way out; the core neither knows nor should. Adding a front door changes nothing in the server — it's a separate module you drop in. Removing one is deleting a file.

Special cases of one mechanism: an inter-branch segment with crypto your regulator mandates; a closed site where only the corporate bus is permitted; a partner channel with a proprietary format; IoT devices on a lightweight protocol. Same core, same permissions, same audit trail — only the adapters differ.

Shrinking the attack surface. A scenario that rarely gets discussed. An identity server has to sit on a public address: browsers, mobile apps, external provider callbacks, back-channel logout — all require a publicly resolvable URL. You can't wrap a tunnel around the whole thing.

But administrative operations — signing-key rotation, forced revocation, bootstrapping the first admin, mutating clients — don't have to be public. Those can move to a separate port, a separate network, and a format of your own: your markers, your protocol version, your authentication.

The effect isn't cryptographic, it's practical: mass scanners and off-the-shelf exploit kits simply don't understand the format. An attacker has to reverse-engineer it before they can produce even a syntactically valid request. Every day of that delay is a day of lead time for your security team. The pattern is old news to banks and the public sector, but it used to require rewriting the stack around a proprietary protocol; here it's one module in front of a standard core.

Mandatory caveat: this is a layer on top of protection, not a replacement for it. Authorization and signature checks stay in place on every operation. A non-standard wire format doesn't substitute for cryptography — it adds information asymmetry to it.

Where we honestly don't cover you

If your platform goes out into regulated open banking — you're a third-party provider, or a bank serving them — you'll need work. Not implemented: mutual-TLS client authentication, rich authorization requests (consent granted for one specific payment to one specific payee), decoupled confirmation through a banking app, and step-up authentication for a transaction.

That's a deliberate boundary, not an oversight. The cryptographic groundwork for all of it is already in the server — proof-of-possession token binding is implemented and covered by tests — so this is a scoped sprint, not a rewrite.

For everything else — your own processing, a wallet, marketplace payouts, corporate treasury, B2B payments, billing — the server covers the job today and in full.

And yes, this works for plain billing too

Worth stating explicitly, because after all the talk about payment cores it's easy to conclude the stack is aimed only at heavy scenarios.

Billing — subscription, telecom, utility, SaaS — lands on our strengths arguably more precisely than processing does:

Pricing changes constantly, and that's billing's central pain. A new plan means new fields and new rules. Where every price-sheet revision normally means a migration and a release window, here it's a class edit and a rollout with charging still running.
Proration is fractions. Mid-cycle plan changes, per-day charges, partial refunds — all multiplication and division, and exactly where rounding error accumulates into invoices that don't reconcile. Precision at the storage layer removes the question.
Recurring charges are a schedule. The scheduler is built in; a charging run is an ordinary route, not a separate service with its own on-call.
Failed payments need retry rules. Dunning with backoff, a dead-letter queue, manual re-run from a support screen — exactly what's below.
Invoices and reconciliation are aggregation, computed where the data lives.

And the part that matters most to a small team: there's no entry toll for complexity. Clustering, the coordinator and the dashboard are optional — the stack runs happily as one process on one server, and everything above switches on by configuration when you grow into it. That's the opposite of enterprise platforms, where you pay for all the complexity up front regardless of whether you need it yet.

Layer three: integrations, where it breaks most and costs most

Financial integration doesn't look the way architecture diagrams draw it. On the diagram: a tidy bus and events. In life: a partner bank that accepts IBM MQ only, because its core was written in the 2000s and nobody is rewriting it. A payout file landing on SFTP at four in the morning. An acquirer on plain HTTP with its own opinion of what idempotency means. Accounting that needs an export on a schedule. And fraud scoring on a queue.

The key idea: integration is won not by whoever has the prettiest bus, but by whoever speaks the counterparty's language. Because you're not going to change the counterparty.

redb.Route ships 27 transports. For comparison, the popular .NET messaging libraries offer four to seven, all of them brokers. They assume you're surrounded by modern services. In finance you are not.

What's actually out there	What covers it
Bank core on IBM MQ	A real transport with transactions, not a hand-rolled adapter
Registries and statements as files	SFTP, FTP, filesystem polling
Acquirers and partner APIs	HTTP, gRPC, WebSockets
Event backbone	Kafka, RabbitMQ, AMQP, Azure Service Bus, Amazon SQS/SNS
Legacy reading straight from a database	Scheduled polling and writes
Notifications	Email, Telegram, push
Clearing and scheduled runs	Built-in scheduler

IBM MQ deserves a note, because it's a maturity tell. It's supported with genuine transactional semantics — including a backout counter and automatic movement of a poison message to a dedicated queue past a threshold. Anyone who has integrated with a bank core knows: without that, one malformed message loops for a day and blocks the channel.

The payment that won't go out twice

The headline technical risk in financial integration is redelivery. The network blinked, the acknowledgement didn't land, the sender retried, the payment went out twice. Resolving that incident means a customer complaint, a manual reversal, and a written explanation.

Three layers of defence, composable:

Transactional processing. Acknowledging receipt to the broker and publishing the result happen together: both or neither. Implemented identically for RabbitMQ, IBM MQ, AMQP and Kafka — so swapping brokers doesn't rewrite the logic.

Persistent duplicate suppression. The operation key is claimed first and confirmed only after successful processing. If the process dies mid-way, the key stays unconfirmed and the operation is correctly retried. State lives in the database, not memory — so it works cluster-wide and survives restarts.

Guaranteed publication. The classic pattern where the event is written in the same transaction as the business operation and published by a separate process.

We get asked why this isn't a single "enable guarantees" switch, the way some competitors package it. The answer is practical. A ready-made guarantees container brings its own table, its own schema and its own notion of how to select unsent items. In finance that table regularly has to live in someone else's database whose schema you don't control, and the selection criterion isn't "where not sent" but a business rule with priorities, windows and caps. A welded-in implementation is a blocker in that situation, not a convenience. Here it's a few lines of configuration where the query is yours, the table is yours, and the transaction is yours.

About international message formats

The question always comes: what about ISO 20022?

There are no shipped codecs for financial message formats — ISO 20022, ISO 8583, SWIFT MT. That's honest, and it's a niche for whoever needs it: transport, framing and schema validation exist, parsing your specific profile is yours. And frankly you'd write it anyway — every bank's ISO 20022 profile differs, and a "ready-made codec" gets modified regardless.

What helps: ISO 20022 is XML with a schema, and validating a message against XSD is a built-in pipeline step. So validating a payment instruction against the official schema is configuration, not development. Same for JSON formats via JSON Schema.

3 a.m.: what a stuck payment costs

Now for the thing that appears in no presentation and determines actual cost of ownership.

A payment is stuck. Not "the system is down" — a down system is visible and gets fixed. Stuck: one operation got an error from an external service somewhere mid-pipeline and never arrived. The customer calls in the morning.

How that goes in a typical project: a developer reads logs, reconstructs what was in the message, hand-assembles a request, resends, and hopes it doesn't double-charge. An hour to a day, requires a developer rather than support, and happens under pressure.

How it goes here:

A route can carry a save-point. Not a log of what happened — a snapshot of the operation's state at a specific place in the pipeline.
If something fails further along, the snapshot lands automatically in a dead-letter store backed by the database — PostgreSQL, MS SQL or SQLite, your pick.
A support operator opens the dashboard, sees the stuck operations with cause and timestamp, and clicks Replay. The operation re-enters its route from the save-point, not from whatever the state decayed into on the way down.

Two details that show this was built for real operations rather than for a feature list:

Capture is opt-in by construction. Only what a developer marked as replayable lands in the queue. The system never claims an operation it wasn't handed — otherwise the dead-letter store becomes a landfill nobody opens.

There's a guard against split ownership of delivery. If a save-point is placed on a route where a broker or a transaction already owns redelivery, the engine warns. That specific mistake — two parties responsible for the retry — is what produces double charges.

Alongside it: a watchdog that classifies routes and raises alerts on hung ones, and in-flight tracking, so you can see which message is sitting in which step right now.

What this means for budget. Triage of stuck operations stops being developer work and becomes first-line work. We put that at 2–4 hours of senior time per week returned to development — but the bigger effect is that a 3 a.m. escalation to a developer stops being routine.

Regulation: guidance in the morning, production in the evening

A financial organisation lives inside a stream of changes it didn't initiate. A regulator publishes guidance. A rate moves. A mandatory field appears in reporting. A partner changes their exchange format. The deadlines aren't yours.

Look at what the path from "guidance published" to "running in production" is made of in a normal project:

Code change — hours.
Database migration, if the structure moved — days of sign-off.
Build and test — hours.
Getting a release window approved — days or weeks.
Service stop, deploy, verification, rollback plan.

Notice: the actual development is item one, and it's the shortest. Everything else is permission to deploy, and it consumes most of the elapsed time.

The same path on this stack:

Change — a new field is a class property, no migration.
The module is built and signed with your key in your pipeline. The key pair is generated with one command; the runtime only ever knows the public half.
It's uploaded — through the management interface or as a file.
The runtime verifies the signature before a single line of code from the package executes. The public key decides, not who put the file there. Unsigned is rejected.
Swap in place: the old version finishes the operations it started, the new one starts alongside, and only after it proves healthy is the old one retired. In a cluster this rolls node by node.

The payment flow never stops. Permission to deploy collapses to "the signature is valid." Rollback is uploading the previous package.

Guidance in the morning, change and build during the day, rollout across nodes in the evening. Not heroics from the on-call shift — a routine procedure.

For the business those are two separate wins: speed of response to a regulator, and the elimination of downtime as a category. In finance the second is often worth more — an approved maintenance window isn't just unavailability, it's a change request, customer notifications, and reputational cost.

The platform team you don't have

There's a layer that teams under fifty people never build. Not from incompetence — it doesn't pay for itself within a single product. The operational envelope: a management interface, a CLI, a dashboard, node coordination, metrics, a watchdog, deployment.

What you get instead is the standard kit: three deploy scripts, log grep, and "SSH in and take a look." Which works while the system is small.

In redb.Tsak, that layer arrives with the product:

Dashboard — 10 screens: cluster overview, a three-level topology tree, per-node detail with live load charts, every route with counters and error rates, per-route drill-down with in-flight operations, the dead-letter queue with a replay button, the hung-route watchdog, log search, access-key management.

CLI — 43 commands. Not just start and stop: sign a module, deploy, validate, roll back, drain a node, return it, rebalance, pull diagnostics, replay a dead letter, pause the scheduler.

Management API — 16 controllers, 45-plus operations, with a typed client. Meaning all of the above embeds into your existing control plane instead of requiring a separate panel.

Observability — Prometheus-format metrics, a ready Grafana dashboard as a file, three distinct Kubernetes probes, a full manifest set including monitoring-operator integration. And, separately, distributed tracing — which deserves its own paragraph.

"Where is my payment" — the question that usually has no fast answer

A customer calls asking where their money is. The operation crossed an inbound request, an authorization check, fraud scoring, a queue, the partner bank and back — five systems, each with its own logs in its own time format.

In a normal project the answer gets assembled by correlating timestamps and fragments of identifiers. Half an hour to a day, and it needs the person who remembers how everything connects.

The right solution is well known — distributed tracing: one operation gets an identifier carried through every system, and afterwards its whole path is one chain with per-step timings. The catch is that building it yourself is a project: pick a standard, thread context across every process boundary and every transport, instrument each connector, stand up a collector, teach the teams.

Here tracing is built into the integration engine and the runtime picks it up on its own. Drop in a module and its routes are already in traces and metrics, with no configuration in application code. Counters for processed and failed operations, a duration histogram, a gauge of operations in flight right now — all present from first start. Export to Jaeger or any compatible collector is one setting; your own spans and counters join the same pipe.

Two engineering details that show this was built for operations:

Collection always runs; export is on demand. Traces are produced regardless of whether export is enabled. So attaching a collector during an incident doesn't require redeploying anything.

Observability may not take the system down, and may not be a hole. Before starting, the metrics exporter probes whether it can bind and, if not, logs a warning and continues without metrics — because an optional subsystem must not take down a payment path. And the collector listens on loopback only; metrics go out through the main API. Telemetry physically cannot end up exposed to the network by accident.

For the business, short version: "where did it get stuck" becomes a matter of minutes and of support staff — rather than a day and a developer who remembers the architecture.

Scaling out is starting another process

Worth a separate note, because it's a direct capital line.

Going from one node to three is the same configuration file on every node, differing only in node id and address. Nodes find each other through shared storage, elect a leader, and the leader distributes modules and rolls updates one node at a time.

No service discovery system, no external coordinator, no service mesh. And not one line of new code.

Going back is symmetric: a node is drained with one command, finishes what it started, and leaves.

An important detail for larger platforms: nodes don't have to be identical. Placement is tracked per module, scoped to a group. So an "acquiring" group can carry one set of modules, "payouts" another, "reporting" a third — all under one coordinator. That lets you separate contours by load and by criticality without splitting them into separate systems with separate operations.

And containers however you like. Three image variants: a headless worker for Kubernetes, a standalone management UI, and an all-in-one for single-server installs. Ready compose sets, Kubernetes manifests, and a no-container archive option. Images are signed. Need a custom build with your own connectors baked in? An ordinary standard .NET build, no proprietary tooling.

What that layer is worth

If you had to build it — component by component, conservatively:

Component	Engineering hours
Management API	250–350
CLI	150–250
Web dashboard	400–600
Node coordination: leader election, registry, distribution	250–350
Zero-downtime updates with graceful drain	100–150
Dead-letter store with replay	100–150
Hung-process watchdog	60–100
Metrics, distributed tracing, health probes	310–480
Images, deployment sets, manifests, signing	210–330
Signature verification for loaded modules	40–60
Total	1,870–2,820

That's over a person-year — before a single line of business logic. And, again, most teams simply don't build it: they live without it and pay later, in night escalations and hours of incident triage.

Hardware: why the same traffic needs fewer nodes

There's a line in a platform's budget that technology posts almost never mention — the infrastructure bill. And it's driven less by traffic than by how much unnecessary work the system does per operation. Two mechanisms here, and they pay out every month rather than once.

Don't rebuild what didn't change

Standard behaviour for nearly any data-access layer: you asked for five hundred operations, so it assembled five hundred objects out of database rows. Next page, then back — assembled again. CPU work producing no value.

In redb every object carries a checksum, and loading happens in two steps. First a cheap query — identifiers and checksums only, no touching field values. Then that's compared against what's already in memory, and only genuinely changed objects get fetched. The rest come from cache without reassembly.

Same on the write side: an object that didn't change never reaches the database.

And this stays correct in a cluster, which is the part that matters most. The checksum comes fresh from the database on every query. So if one node changed an operation, the next node sees the mismatch on its next request and re-reads it by itself. No separate cache server with its own operations and its own bill, and no invalidation broadcast between nodes — the mechanism that breaks most often and most quietly in distributed systems.

The mode is a setting, and the safe one — with validation — is the default. A monolith can turn validation off and go faster still; a distributed deployment leaves it on and gets correctness for free.

Don't hold a thread just to wait

The second mechanism is about what a payment system spends its time doing.

Which is waiting. For the acquirer's response. For the bank's confirmation. For the fraud verdict. For a disk write. Actual computation is a fraction of a percent.

In a synchronous model each of those waits pins a thread. Threads are memory and context switches, and they run out considerably earlier than CPU does. Hence the familiar picture: the server is at fifteen percent utilisation and requests are already queueing.

The routing engine is async by construction: its core holds around a hundred and fifty async methods against eight blocking calls — and all eight sit in synchronous convenience wrappers, not in the processing pipeline. While an operation waits on an external system, the thread serves others.

Practical meaning: one node holds substantially more concurrent operations on the same CPU. For a payment gateway, where waiting is nearly all of the elapsed time, that's not a difference measured in percent.

What that means in money

Resource savings convert down a chain, and every link is its own budget line:

Fewer cores per node — and enterprise database licenses, plus a chunk of commercial software, are priced per core.
Fewer nodes for the same traffic — a direct cloud or hardware bill.
The standby environment mirrors production, so any saving on the primary automatically doubles.
Less load on the database itself — and in a financial system the database is usually the most expensive component and the hardest to scale: you add an app node, you don't add a database.

We're deliberately not quoting percentages here. They depend on your load profile, and any tidy figure in a post like this would be a figure from nowhere. But the mechanism is verifiable, and you'll see it on your own load immediately.

The important part is that this saving is recurring. Saved engineering hours are a one-time gain. The infrastructure bill arrives every month, for the life of the system.

People: why one stack is cheaper to run

There's an effect that never appears in a budget and in practice costs more than migrations.

Take a normal .NET project two years and fifty features in. What you'll find: two controller styles, because the approach changed. Three validation approaches. Two background-work mechanisms — the scheduler adopted at the start, and the second one added when the first didn't fit. A hand-rolled mapper next to a library one. And a data-access layer half the team bypasses because it's faster that way.

Nobody is surprised, because this is what always happens when the scaffolding is chosen fresh on every task.

Our stack constrains that freedom deliberately. Three rules total: a data structure is a class, an entry point is a route, an integration is a connector. Not out of hostility to variety, but because variety in the foundation is paid for forever.

In money:

Onboarding. A new developer learns one approach rather than five that "grew historically." Difference: 3–5 days per person. On a team of ten with normal attrition, 150–250 hours a year.

Review. The "but how should this be done" argument doesn't arise, because there's one shape. Half an hour to an hour per pull request, a hundred and fifty of those a year — another 75–150 hours.

The refactor that never happens. In a heterogeneous codebase, every eighteen months to two years someone launches a "let's unify everything" initiative worth 200–400 hours that gets half-finished. Here there's nothing to unify.

Key-person risk. No longer about hours. When the scaffolding is uniform, a departing lead doesn't take with them the knowledge of why it's done one way here and another way there. For a financial organisation, where team turnover is a genuine operational risk, that's a line of its own.

Procurement: what developers don't think about and directors do

Three questions that surface at sign-off and can stop a project after the technical side has satisfied everyone.

Licenses. Every Pro capability — optimised query generation, partial writes, parallel materialisation, clustering, advanced analytics — is free across the whole 3.x line. No key, no license server, no registration, commercial production included. Not "free up to a volume", not "free for development." Free. Versions you're already running stay free permanently.

The open portion is Apache 2.0. Pro packages are closed-source but free.

Source code. Companies that adopt the ecosystem can request the Pro sources — for a security audit, escrow, or in-house builds. In enterprise procurement, source escrow for a proprietary component isn't a bonus, it's a line in the requirements, and it usually costs separate money and separate negotiation.

No vendor lock-in. Data sits in PostgreSQL or MS SQL — your database, your infrastructure, your control. The storage structure is open and documented, so the database reads with ordinary SQL, without our code and without our tools. There's a first-class export and import for the whole database, including across engines. No closed formats anywhere. The worst case — "the vendor disappeared" — leaves you with a running system, your data in readable form, and, if needed, the sources.

For contrast: the typical alternative for the facade layer is either a commercial license priced at enterprise tier for a financial institution, or an open product on a foreign stack that needs an operations team of its own.

The whole bill

Everything above, in one table. This is a model with stated assumptions, not a measurement of two parallel teams — but the assumptions are conservative, and every line can be argued with individually.

One-time cost that doesn't occur:

What you don't write	Engineering hours
Transports — the 9 a payment platform actually needs, at production quality	900–1,350
Integration patterns — duplicate suppression, splitting, aggregation, routing, retries, circuit breaker, compensation	400–800
Operational layer — API, CLI, dashboard, clustering, observability, deployment	1,870–2,820
Identity facade — adoption, client management, audit, MFA, federation	600–1,000
The data-model design phase that never happens	440–880
Testing product hypotheses — 10–15× cheaper per test	~560
Total	4,770–7,410

That's 2.8–4.4 person-years.

Recurring cost that doesn't occur:

What	Hours per year
Database migrations	360–600
Preparing and running release windows	~450
Triaging stuck operations with developer time	100–200
Maintaining your own connectors as external systems change	150–250
Onboarding and review on heterogeneous scaffolding	225–400
Total	1,285–1,900

That's 0.8–1.1 of a permanent headcount.

Rates vary too much across markets for a single number to be useful, so substitute your own. Fully-loaded cost per engineering hour — salary plus taxes, benefits, workspace, management, holidays and idle time, not the number in the offer, which is the most common mistake and understates the result two-to-threefold:

Fully-loaded hourly	One-time (4,770–7,410 h)	Annual (1,285–1,900 h)
$60	$286k – $445k	$77k – $114k
$100	$477k – $741k	$129k – $190k
$150	$716k – $1.11M	$193k – $285k

Plus no license fees, and source escrow already answered.

One honest caveat: calling this a saving is only fair where the team would genuinely have built all of it. An organisation that would have bought instead gets the benefit in another form — absence of license fees and absence of an integration zoo. And hours spent on your business logic — the accounting model, pricing, fraud rules, reconciliation — nobody saves and nobody should. What gets saved is the infrastructure layer underneath them.

Where we're a bad fit

The section that usually doesn't get written. We'll write it, because understanding the boundary is cheaper at the start than halfway in.

The accounting model is yours. No double-entry, no chart of accounts, no posting rules, and none planned. We give the foundation, you describe your accounting. If you were shopping for turnkey processing, this isn't it.

Regulated open banking needs work. If the platform faces outward as a third-party provider or as a bank serving them, you'll need a sprint across the standards listed earlier. The groundwork is there and the scope is knowable, but it's not in the box today.

Financial message codecs are yours to write. Transport, framing and schema validation exist; parsing a specific ISO 20022 or ISO 8583 profile is yours.

No multi-region clustering out of the box. Coordination assumes low latency to a shared database. The pattern for geographic redundancy is a cluster per region, federated a level above.

Horizontal sharding is a kit, not a feature. Every necessary mechanism is in the store: block key generation with the ability to move the key source to a separate database, per-connection cache isolation, several independent stores in one module. An assembled "switch it on and go" solution isn't there yet. That's a post of its own, and it's in progress.

Module isolation is not a sandbox. A loaded module runs with the process's privileges: trust comes from the signature at the door, not from restricting rights after start.

For enterprise practice that barely matters. The contractor delivers source, the build runs in your pipeline and is signed with your key — the pair is generated with one command, and the verifying side only knows the public half. Then the signature doesn't mean "we trust the contractor", it means "this is exactly the code that passed our review and our build", which is stricter than any sandbox around a third-party binary.

The real limitation remains for one scenario: a module arriving as a finished binary from a supplier you don't control and whose code you haven't seen. There you'll want OS-level isolation — a process or container per module.

Wrapping up

You can't buy a payment platform ready-made — the part you actually get paid for is always written for you. But it's the smaller part of the code and the smaller part of the budget.

The larger part is the transport to a bank that speaks IBM MQ. It's the defence against double-charging. It's an authorization server that will survive an audit. It's a dashboard that shows where an operation stopped. It's deploying without pausing payments. It's node coordination once one node stopped being enough. And that's roughly three person-years plus a standing headcount — if you write it yourself.

Our bet over the last four years is that this layer should be shared. One stack across all three parts of the platform, one version across the ecosystem, one set of rules for the whole team. A store that counts money without loss and changes its schema without migrations. An integration engine that speaks the counterparty's language instead of demanding they speak yours. An identity server checked against an external reference. A runtime that ships changes without a maintenance window and hands support a button instead of a phone call to a developer.

And all of it free across the whole 3.x line, with sources on request.

If you're working on something similar — tell me in the comments which of these hurts most. We have detailed technical write-ups per layer, and the next ones will follow real questions rather than what we happen to find interesting.

Sources and releases: github.com/redbase-app. About the store: redbase.app. Previous posts in the series: my dev.to profile.

If this was useful — a ⭐ on GitHub helps others find it.

redb 3.4.0: day-two operations for a .NET stack — replay what failed, hot-patch the framework, lock down the control plane

rinat kozin — Mon, 27 Jul 2026 21:48:33 +0000

Building a system and running one are two different engineering problems. The first is done when it holds up under load. The second starts with the questions someone asks while holding a pager: what died overnight, and how do I re-run it? who hit force-stop? can I ship a one-line library fix without rebuilding the entire runtime? why is a service-account password sitting in plain sight on a dashboard page?

Previous releases of our stack answered the first question. 3.4.0 is entirely about the second.

Quick refresher on the cast: redb, a typed store on top of Postgres/MSSQL/SQLite; redb.Route, an integration engine in the Apache Camel spirit with 30+ connectors; redb.Tsak, a runtime with a dashboard, hot-reload and clustering; and redb.Identity, an OIDC / OAuth 2.1 server. All of it runs in our own production and ships as packages, images, and standalone archives.

Four things landed in 3.4.0, and every one of them is about the day after the deploy:

Replay checkpoints — a save-point inside a route, so the tail can be re-run later from known-good state. This one goes all the way through: a primitive in the engine, a dead-letter queue in the runtime, a button in the dashboard.
A shared runtime layer — the framework now lives beside the app instead of inside it. Patching a library is a file swap, not a release of everything.
Secrets out of the logs — declarative redaction of endpoint URIs, plus named connection factories so the credential never enters the route to begin with.
Roles and signed modules — role enforcement on the management API, a persistent audit trail, and a cryptographic trust anchor for code the runtime loads into its own process.

Quietly but importantly: redb.Identity moved onto the ecosystem version (1.2.2 straight to 3.4.0). One number across the stack, so "which Identity works with which Route" is a glance instead of a lookup table.

And as of 3.3.0, still true here: every Pro package is free across the whole 3.x line. No keys, no license server, no sign-up. dotnet add package redb.Postgres.Pro and it works.

Part of the redb / redb.Route series — recent posts first:

The Telegram connector: a bot as a route

redb 3.3.0: an enterprise .NET stack you actually own

Leaving MassTransit for a Camel state of mind: the Kafka connector, Scatter-Gather, and transactions

Apache Camel for .NET, dissected: the HTTP connector with no ASP.NET MVC + the Content-Based Router

redb.Route — Apache Camel for .NET: 22 transports, 30+ EIP patterns, compiled DSL

Sources: github.com/redbase-app. About the database itself: redbase.app.

Part 1. Replay: a save-point inside a route

Starting with the feature I'm happiest about — because it runs through all three layers and looks natural in each one.

The problem everyone solves by hand

A five-step route. Step three charged the card. Step four blew up because the receipt service was down for ten minutes. Now what?

None of the usual answers are good. Retrying the whole route charges the card twice. Retrying from the exchange's current state is worse than it sounds: the body has already been reshaped by the steps in between and headers have been overwritten, so you're not replaying what happened — you're replaying whatever it decayed into on the way to the failure. Picking through it by hand at 3 a.m. isn't an answer at all.

The right model is a save-point: freeze the message exactly where things were still good, then re-run the tail of the route from that frozen state.

What it looks like in the DSL

From("timer://poll?period=5000")
    .Process(chargeCard)
    .Replayable("after-charge")      // save-point: the card is already charged
        .Process(sendReceipt)
        .To("http://receipts")
    .EndReplayable();

Everything after the marker is the tail. Each exchange passing through is snapshotted into exchange.Properties["route.checkpoint"] (last marker wins). There's a lambda form too, if you'd rather not track matching End calls:

.Replayable("after-charge", r => r
    .Process(sendReceipt)
    .To("http://receipts"))

Replay is a typed in-process call:

await context.ReplayAsync(routeId, "after-charge", snapshot, ct);

Declare the marker with exposed: true and it's additionally published as direct:__replay:{routeId}:{name}, so another route can simply .To(...) it. The default is private: a save-point shouldn't accidentally become a public entry point into the middle of your route.

`Snapshot()` — and why it isn't `Clone()`

This needed a new primitive in the engine, and it's the most interesting part.

IExchange has had Clone() for a long time. But Clone() deliberately shares the body with the original — Splitter aggregation depends on it, because the branches need to see the same object. For a save-point that's exactly backwards: if the body is shared, the next step mutates it in place and your "frozen" state was never frozen at all.

So IExchange.Snapshot() / IMessage.Snapshot() landed: a genuinely deep, isolated copy. v1 handles immutable bodies, byte[], and ICloneable, and throws loudly on anything else rather than quietly doing a shallow share. That part isn't negotiable — a silently shared body inside a save-point mechanism is the kind of bug that surfaces a month later, in production, on the one exchange you needed.

A snapshot carries no DI scope: it's dormant data headed for a database row, not a live exchange, so there's no per-message leak.

While we were in there, we also fixed the doc comment on Clone(), which had said "deep copy" for years while never being one. Both methods now describe what they actually do.

Where a checkpoint may and may not go

A save-point can't cross a branching composite (Choice, TryCatch) — the tail stops being well-defined, because which branch is the tail? Inside a long-running transaction it's allowed but warrants a warning: replay happens outside the original transaction.

We could have hard-coded a couple of ifs in the validator. Instead there's now a general mechanism, because constraints like this keep showing up:

ICompositeScope / IDurableScope — scope category markers;
IScopeNestingRule — a node declares Allowed / Warn / Forbid against an ancestor category;
IBranchingDefinition — definitions whose children live outside Outputs (Choice When/Otherwise, TryCatch catch/finally) expose them for a generic tree walk.

The validator applies the rules uniformly: Forbid is a build error, Warn goes to the log. A new structural constraint now ships on the definition, never in the validator — which is precisely what Camel-style engines tend to get wrong, letting constraints sprawl across code that has no business knowing about them.

Layer two: the dead-letter queue in Tsak

A primitive in the engine is nice, but the person on call doesn't want an abstraction — they want the list of what failed overnight. So Tsak 3.4.0 puts a DLQ with replay on top of checkpoints.

CheckpointDlqHandler is a Tier-3 error handler installed on every module context. On failure it reads route.checkpoint off the exchange and dead-letters it. The key property: opt-in by construction. Only routes carrying a .Replayable() marker leave a checkpoint, so only those get captured. The queue never fights a broker or a transaction that already owns redelivery — it picks up exactly what you explicitly marked as "this needs a human re-run."

From there it's ordinary operational plumbing:

tsak_dlq — a flat table (PG / MSSQL / SQLite), created at startup;
ExchangeSnapshotCodec — serializes the snapshot for durable storage. byte[] and string round-trip exactly; everything else goes through System.Text.Json with the exact CLR type restored when its assembly is loadable. A non-serializable body is stored visible but flagged not-replayable — capture never breaks outright;
API: GET /api/exchanges/failed (filtered, paged), POST /api/exchanges/{id}/replay, DELETE /api/exchanges/{id} — role-gated and audited;
CLI: tsak dlq list | replay | discard;
a Dead-letter dashboard page with server-side filtering and paging, so a large table never gets dragged into the browser;
retention as a first-class cron://tsak-dlq-retention route on the system context (Tsak:Dlq:RetentionDays, default 30). Not a hidden timer buried in the runtime — a normal route, visible in the API, the dashboard, and the scheduler page. We eat our own cooking.

On semantics, plainly and without marketing: this is at-least-once and operator-driven. The tail may run more than once, so replayed side effects need to be idempotent. It is not Temporal-style durable execution and doesn't pretend to be. It's a tool for the person on call: show me what broke overnight, and re-run it once the fix is in.

Part 2. The shared runtime layer: patch the framework without a rebuild

The second big item is architectural, and it's about how fast you can react.

Before

Tsak takes your routes (.tpkg modules) and turns them into a service with a dashboard, metrics, and clustering. The framework itself — redb.Core(.Pro), the providers, redb.Route.* — used to sit in the app's bin folder. Which meant a patch to any library required rebuilding and re-releasing everything. One fix in the Kafka connector: rebuild Tsak, re-cut the archives, re-push the images, re-run the whole verification pass.

After

The framework moved to Libs/shared/ and is loaded from there at startup. Only redb.Tsak.* and redb.Licensing stay in the bin.

worker/
├── redb.Tsak.Worker.dll        ← the application
├── redb.Tsak.Core.dll
└── Libs/shared/
    ├── redb.Core.dll           ← the framework — swappable
    ├── redb.Postgres.Pro.dll
    ├── redb.Route.dll
    ├── redb.Route.Kafka.dll    ← connectors too
    └── runtimes/…              ← native dependencies alongside

The payoff: a binary-compatible patch to any leaf, provider, or connector — or a brand-new beta connector — ships by swapping a DLL in Libs/shared/. No rebuild, no re-spin of the Tsak and Identity archives. A 3.4.0 → 3.4.1 framework patch becomes a file copy.

How it works, briefly:

Early bootstrap. SharedRuntime.InstallEarly is the very first statement in Program.cs, before a single redb type is touched. It installs the shared-layer resolver and byte-loads the framework — the file is never locked, which is exactly what makes it swappable on a live box. It reuses primitives we already had: tracker unification (so .tpkg modules still see one identity per redb type), a per-assembly native resolver (librdkafka, e_sqlite3), and version-tolerant forwarding.
Fail fast. A missing or corrupt framework DLL in Libs/shared/ aborts startup immediately, naming the assembly and where it looked. Infinitely better than a MissingMethodException a day later under load.
Compat gate. At startup the shared redb minor is checked against this Tsak build's minor. Patch drift is allowed — that's the entire point — but a minor mismatch, or a mix of minors inside Libs/shared/, stops the process.
GET /api/system/assemblies (admin) — what redb is actually loaded: name, version, and origin (shared / bin / runtime). The diagnostic counterpart to a swappable layer, answering "which redb is running right now."

Tooling collapsed to one manifest (scripts/shared-manifest.psd1) and one parameterized scripts/build-shared.ps1 that serves both dev and publish. Plus a new scripts/refresh-shared.ps1: rebuild one library and drop its DLL into a target Libs/shared/ — your dev tree or an unpacked archive. The "patch without rebuilding Tsak" flow, as an actual command.

We verified it rather than assuming: 0 framework assemblies in the bin root vs 12 in Libs/shared; a clean start loads all 12 from the shared layer and boots (SQLite Pro, cluster, scheduler); Kafka with librdkafka, Mail with MailKit, and SQLite with e_sqlite3 all served through the shared layer and exercised end to end; .tpkg modules unify; the negative case aborts with a clear message.

Planned maintenance: cordon / uncordon

Same "day after the deploy" theme, and it's in Pro — so, free.

A cluster node used to have two states: working, or gone. That's not enough for a rolling upgrade: rebalance reshuffles everything and remove-node is a hard eviction, when what you want is the middle state — this node finishes what it's doing and takes on nothing new.

tsak cluster cordon   node-3     # takes no new work, drains its route locks to peers
# … upgrade the node …
tsak cluster uncordon node-3

The Cordoned flag is durable and orthogonal to status: the node stays Online, because it is alive — it's just closed for new business. The per-route watch loop reads a process-local mirror of the flag, refreshed from the node's own record on each heartbeat, stops acquiring route locks, and releases the ones it holds so peers can pick them up. Work in flight drains normally. There's an API (POST /api/cluster/nodes/{id}/cordon and /uncordon, audited), a client method, the CLI above, and Cordon/Uncordon buttons on the dashboard's Cluster page.

If you've spent time around Kubernetes, both the semantics and the word will look familiar. That's deliberate: don't invent new vocabulary where the industry already has one everybody knows.

Part 3. Secrets: don't mask them, don't accept them

The third block is operational security — and the interesting part isn't what got fixed, it's how the approach changed.

Declaration beats guesswork

In an engine where endpoints are URIs (ldap://…?bindPassword=…), sooner or later you have to decide which parts of that URI may reach a log. The classic answer is a deny-list of parameter names: password, secret, apiKey…

A deny-list fails open. Add a new option carrying a credential and it goes to the logs in the clear until somebody remembers to extend the list. The failure is silent: you find out by reading logs, not by failing a build.

In 3.4.0 the source of truth is a declaration on the option itself:

public class LdapEndpointOptions : EndpointOptions
{
    public string Server { get; set; } = "localhost";   // printed in logs

    [Sensitive]
    public string? BindPassword { get; set; }           // always ****
}

EndpointOptions.BindFromUri harvests those declarations by reflection (once per options type) and feeds them to the URI sanitizer. The set of secret keys is derived from the code and never hand-maintained — a new credential option can't be forgotten, because the only thing you could forget is the attribute, and the attribute sits right there on the property.

This is exactly how Apache Camel does it: @UriParam(secret = true) is the declaration, and the runtime list is generated from those annotations by a build plugin. The .NET version needs no build step — reflection is enough. All 37 credential options across 22 connectors are annotated. The name-based heuristic survives as a backstop for a URI rendered before any endpoint of that scheme exists.

Formatting is preserved: the new EndpointUri.Sanitize(string) keeps the scheme, ://, path, parameter order, and every non-secret value byte for byte, replacing secrets with a constant ****. Every boundary now goes through it: route-build and endpoint-start logs, the redb.route.endpoint OpenTelemetry span tag and metric label, in-flight exchange and health-check metadata, and the CompiledRoute.FromUri DTO the CLI and dashboard render. Unnamed routes now derive a sanitized route id too, so a secret can't leak out through a {RouteId} log line.

Also in this pass: userinfo passwords are finally inside the fence (amqp://user:pass@host → user:****@host); redb.Route.Elasticsearch sanitizes node URLs in its startup log; and redb.Route.Exec logs the executable plus an argument count — command-line argument values are secrets far too often to log on principle.

There's a public API for connector authors as well: EndpointUri.Sanitize(string), IsSensitiveKey(string), and AddSensitiveKeys(params string[]) — the direct analogue of Camel's addSanitizeKeywords.

Separately, EndpointUri.RedactSecrets(string) handles arbitrary text: a driver exception is perfectly capable of dragging a connection string with Password=… into ex.Message, and OnExceptionProcessor logs that message on redelivery and on retries-exhausted. Both sites now run it through the redactor. One honest limitation worth knowing: with LogStackTrace enabled, the exception object itself is handed to the logger and can't be scrubbed in-process — that path needs a filter in your logging sink.

Better than masking: never take the secret

Masking is defense. Offense is making sure the credential never enters the URI at all. So 3.4.0 rolls out a named ConnectionFactory across the ten connectors that had no such mechanism: Telegram, MQTT, HTTP, Mail, FTP, SFTP, SignalR, gRPC, TCP, WebSocket.

context.AddToRegistry("support-bot", new TelegramConnectionFactory {
    Token = Environment.GetEnvironmentVariable("TELEGRAM_TOKEN")! });

r.From("telegram://receive?connectionFactory=support-bot")   // no token in the route

No secret in the URI means nothing to mask in logs, telemetry, or the dashboard. The factory fills in only the options the URI didn't set explicitly, so an inline URI value always wins and existing routes are untouched; a name that isn't in the registry logs a warning and falls back to URI parameters.

One detail I'm prouder of than the feature itself. For connectors whose address lives in the endpoint path (HTTP, Mail, SignalR, gRPC, TCP, WebSocket) the factory deliberately carries no host or port. Otherwise an innocent typo in a factory name could silently redirect a route somewhere else. For the same reason the wss scheme still forces TLS on regardless of what the factory says. A mechanism for credentials must not quietly become a mechanism for routing.

redb.Route.Ldap got an LdapConnectionFactory in the same pass — and with it, a route that carries no credentials whatsoever:

context.AddToRegistry("honest-ldap", new LdapConnectionFactory {
    Server = "ldap.corp.local", Port = 636, Ssl = true,
    BindDn = "cn=svc-reader,dc=corp,dc=local",
    BindPassword = Environment.GetEnvironmentVariable("LDAP_BIND_PASSWORD") });

r.From("ldap://SEARCH:dc=corp,dc=local?connectionFactory=honest-ldap&filter=(objectClass=user)")

One caveat to internalize: CompiledRoute.FromUri is now a display value. Routing identity, endpoint cache keys, and message flow didn't change by a single byte, but parsing FromUri to recover credentials is no longer a thing you can do. It's a display, not a data source.

Part 4. Who can do what: roles, signed modules, audit

The management API is, functionally, the keys to prod. In 3.4.0 the keys got locks.

The role model

Tsak API keys have carried roles since 1.0.0. In 3.4.0 those roles started being required at the endpoints — until now the viewer < operator < admin ladder existed in the model, but no endpoint declared a requirement, so a key issued for dashboard access could do everything an admin key could.

It's declarative, in the same style as everything else here:

RequiresRoleAttribute — on an action or a whole controller; multiple roles are OR-ed; a method-level attribute overrides the controller-level one;
NoRoleRequiredAttribute — for technical endpoints that must never start answering 403;
TsakRoles — the ladder itself, with reader/ops synonyms; custom roles match by exact name only;
RoleAuthorizationProcessor — the enforcement, wired into the system pipeline immediately after a successful auth check. It resolves the target action through the same ControllerRegistry the dispatcher uses, so both agree on where the request would have landed.

How it shook out: admin covers all of /api/auth/* (reads included), /api/users/*, deleting a context or a module, route force-stop, cluster rebalance and node removal. operator covers diagnostics and logs (both expose internals) plus every mutating endpoint by default. viewer covers every other GET.

A specific concern here was not breaking probes. The check runs only for authenticated exchanges, so auth-exempt Kubernetes probes pass straight through; HealthProbeController is additionally marked [NoRoleRequired], so even if an operator narrows Tsak:Api:AuthExempt, the probe can't start returning 403 and take the deployment down with it. The echo and Prometheus routes have their own pipelines and never reach the check at all.

Compatibility is staged, as it has to be for something like this. Nothing changes when auth is disabled. Keys with no roles keep full access and log a one-time warning — then, once every key has been reissued with explicit roles, you set Tsak:Auth:RolelessKeysAreAdmin=false and close the door. Enforcement as a whole can be switched off with Tsak:Auth:EnforceRoles=false.

Coverage: 32 tests on the processor and the ladder, plus 27 integration tests driving the real Kestrel pipeline with viewer, operator, and roleless keys — including proof that probes answer without a key and never return 403.

A module is code. So: signatures

Deploying a module used to mean filesystem access. Now modules can be uploaded and rolled back over the API — but since a module is code Tsak loads into its own process, the whole feature is built around a trust anchor:

tsak module keygen              # generate an ECDSA key pair
tsak module sign  my.tpkg       # → my.tpkg.sig
tsak module validate my.tpkg    # dry run: same checks as upload, installs nothing
tsak module deploy my.tpkg      # upload (signature in a header)
tsak module rollback my-module  # restore the previous version

The parts that matter:

Upload is off by default (Tsak:Modules:Upload:Enabled=false) — a node that doesn't need remote deploy exposes no upload surface at all.
ModuleSignatureVerifier does RSA/ECDSA detached-signature verification with the BCL only. No cosign dependency.
Enforcement at the load boundary: with Tsak:Modules:Signature:Required=true and a configured public key, every .tpkg — uploaded or dropped into the directory by an operator — must carry a valid .tpkg.sig or it's refused before any of its code loads. The trust anchor becomes the public key, not filesystem access. That's stricter than the WSO2 MI default.
Upload-time guards: a size ceiling, valid-ZIP and manifest checks, the stored name taken from the manifest and sanitized (no /, \, .. — path-traversal and zip-slip safe), fail-fast signature verification, atomic install (temp → move), previous version archived.

Related: staged validation before hot-swap. A package update used to tear down the running version before opening the new one — so a broken package left the context with no modules at all. Now the new .tpkg is first opened in a throwaway collectible ALC (without mutating the shared assembly tracker, so running modules are untouched) and checked that it loads and discovers at least one module. Only then is the old version torn down. A package that won't open, or that has no modules, is refused with a logged reason — and the current version keeps running.

An audit trail that survives a restart

Admin actions used to go to the log, with lifecycle history in a 1000-entry in-memory ring — so after a restart there was no record of who did what. Now there's tsak_audit_log: a flat table (deliberately not a redb object — it's append-only, it grows, and its schema is fixed), created at startup for the configured provider, exactly the way the Quartz tables already were.

It's written through a direct://tsak-audit route (Sql.Execute INSERT) — and that's a feature, not showing off: because the sink is an endpoint, the same event stream can be pointed at a file, a broker, or an HTTP collector by configuration alone, with no new code. Writes are fire-and-forget through a bounded queue with a background pump: an API call never waits on the database, a broken audit backend can never take the node down, a backend failure falls back to the log sink, and sustained flooding drops the oldest queued events with a warning.

Reading is GET /api/audit (role admin, filtering and paging entirely server-side), the tsak audit CLI command, and a new Audit dashboard page. Retention is — again — a normal cron://tsak-audit-retention route (Tsak:Audit:RetentionDays, default 90; 0 keeps forever).

For deployments with no database, audit stays in the log but now emits a [tsak-audit]-anchored line with one JSON object right after it, so a standalone node can be grepped and parsed without any log-format wrangling.

Get paged, don't poll

Last of the operational block: the watchdog detected hung and suspected exchanges, but the alerts just piled up behind GET /api/watchdog/alerts — a poll nobody performs at 3 a.m. Now they're pushed.

AlertDispatcher fans each new alert out to every enabled channel, fire-and-forget (bounded queue plus a background pump, same as the audit sink): the watchdog scan never blocks on a slow SMTP server. Dedup is by context + route + exchange + level within DedupWindowMinutes — the scan rebuilds its snapshot every cycle, so without it a single hung exchange would page you on every tick.

Channels, all off by default: webhook (Slack / Teams / PagerDuty / any collector), telegram (Bot API over plain HTTPS, no connector), email (SMTP via the BCL), and endpoint — the generic one: send to any redb.Route producer URI (kafka:, rabbitmq:, amqp:, sqs:, mqtt:…) through a ProducerTemplate. That last channel covers every broker with zero per-connector code, and because the host supplies the component, no broker ever becomes a compile-time dependency of Core.

You can verify the wiring without waiting for a real outage: POST /api/watchdog/test-alert sends a synthetic alert through every enabled channel and returns the per-channel outcome, bypassing the dedup window. In the dashboard it's a "Send test alert" button on the Watchdog page's Alert Delivery panel.

Part 5. The core: a scheme name becomes your decision

Down to the storage layer. The headline change is small on paper and very visible in practice.

`[RedbScheme(Name = "...")]`

A scheme used to be named after the CLR type's FullName, with the string in the attribute serving as a cosmetic alias. The consequence is familiar: a scheme's identity in the database was welded to your namespace and class name. Rename the class or move the namespace, and you either don't refactor or you go fix the database by hand.

[RedbScheme("Customer note", Name = "Notes.Note")]  // positional arg is still the alias
public class Note { … }

The positional argument is, and stays, the alias — an explicit name is only settable through the named Name parameter, so not one existing declaration changes meaning, and types without Name behave exactly as before.

On the first sync, a type with an explicit name has its scheme renamed in the database. The lookup walks a three-step chain — explicit name, FullName, short type name — and the first match is renamed in place. Physically that's a single-row UPDATE of _schemes._name: the id is preserved, objects, structures and values are untouched, and polymorphic loading (which resolves through scheme_id) never notices.

The honest warning, which is in the changelog and belongs here too: renaming requires updating every consumer of that database together. An app version that predates the explicit name won't find the scheme under its new name and will create a second one, silently splitting objects across both. If that has already happened, redb now detects it and refuses to continue rather than picking one at random.

Names are validated against C# identifier rules (Latin letters, digits, _, ., +; no reserved words; 128 chars max) in C#, before any SQL is issued, so the error names the offending type. And they're all validated up front and reported together in a single AggregateException — fixing a codebase one name per restart would have been cruel. Human-readable titles still live in Alias, which is free-form.

Provider parity got closed in the same pass: MSSql and SQLite now carry a _schemes name-validation trigger mirroring the PostgreSQL rule for rule, so a name accepted by one provider is accepted by all three. And a scheme's _alias now syncs on every sync (structures always did): the attribute is the source of truth, removing it resets _alias to NULL.

Load checks the scheme

LoadAsync<TProps> now verifies the object's _id_scheme against the scheme TProps maps to — before anything reaches the cache.

By default a mismatch returns null. That's not an arbitrary pick: a soft-deleted object gets scheme -10, and soft-delete callers expect exactly null back. If you'd rather have a genuine type mistake be loud, set RedbServiceConfiguration.ThrowOnSchemeMismatch = true and get a RedbSchemeMismatchException instead. The untyped LoadAsync(objectId) is unaffected either way.

Several nodes starting at once

A scenario reproduced on a three-node cluster: several instances start against a database that doesn't have a given scheme yet, all of them miss the lookup, and all of them issue INSERT INTO _schemes. Exactly one wins.

Scheme creation now uses a conflict-free statement per dialect (ON CONFLICT (_name) DO NOTHING on PostgreSQL and SQLite, INSERT … WHERE NOT EXISTS with UPDLOCK, HOLDLOCK on MSSql), and whoever loses the race reads back the winner's row.

The detail that makes this more interesting than it looks: catching the unique violation wouldn't have worked. On PostgreSQL a failed statement inside a transaction poisons it, so the follow-up read fails with 25P02. The right answer isn't "handle the exception" — it's "don't create the conflict." Both creation paths are covered, typed and untyped.

Pro data migrations now run on every provider

The Pro data-migration mechanism works across all three providers and is covered by tests (MigrationTestsBase: apply, history row, idempotency, dry run, against all three Pro fixtures). Getting there sorted out four separate things: how the executor obtains the DB context; dialect-aware UPDATE generation (ISqlDialectPro.Migration_UpdateTarget — SQLite forbids aliasing an UPDATE target, T-SQL wants the alias bound in a trailing FROM); a SQLite-native _migrations DDL (with a REAL Julian _applied_at, per SQLite's own time convention in redb); and dropping IDENTITY from _migrations._id on MSSql, since the executor supplies ids explicitly like everywhere else in redb.

Existing databases don't pick up the new _migrations DDL — initialization is skipped when _schemes already exists. In practice that breaks nothing, since there's no migration history there to preserve.

Small things you'll feel

Hot-reload no longer pins memory. The process-wide schemeName → Type index held strong Type references, and a Type keeps its AssemblyLoadContext alive — so a collectible ALC (the thing Tsak's hot-swap is built on) could never be collected, and after a reload the stale instance produced a false name conflict against the fresh one. Entries are now WeakReference<Type> with dead ones pruned on lookup, and two instances of the same FullName from different ALCs are recognized as a reload rather than a clash. Distinct types sharing one explicit Name still conflict, by design.
redb cache options are reachable from Tsak config — the Tsak:Redb:Cache section (props cache and TTL, list cache, metadata cache, AutoRecomputeHash, cache domain). A Tsak node used to run on redb's defaults because only two settings were passed through. Every key is optional and an absent section changes nothing, and the whole set with default values is spelled out in appsettings.json so the knobs are visible and editable in place. Enabling SkipHashValidationOnCacheCheck together with clustering logs a startup warning: trusting the cache without re-checking the object hash is single-writer territory.
679 lines of dead cache-interface layer removed — five interfaces that referenced only each other, with no implementations and no consumers. The live caches are GlobalMetadataCache, GlobalListCache, GlobalPropsCache, and the type index. Technically a breaking change since the types were public; practically unusable, since nothing implemented them. The caching README.md was rewritten to describe the caches that actually exist.

Part 6. redb.Identity: one version, JAR, and an honest conformance answer

Why 1.2.2 → 3.4.0

Until this release redb.Identity rode its own 1.x line while core, Route, and Tsak moved together on 3.x. Two numbering schemes turned "which Identity works with which Route" into a lookup instead of a glance.

From 3.4.0 on, Identity shares the ecosystem version and jumps straight to the shared number — the same redb.Route 3.4.0 it's built against. Every future ecosystem release bumps it too.

To be explicit: this is a version realignment, not a semver-breaking release. There are no breaking API changes here, and the one new feature is off by default. The major digit changed because the number is now the ecosystem's, not because Identity's contract broke. The old 1.0.1–1.2.2 tags remain valid history.

JAR (RFC 9101) — signed authorization requests

/connect/authorize now accepts a signed request object, gated behind Features.EnableJar and off by default, so every prior release behaves identically (a request parameter still gets request_not_supported).

When it's on: a signed request (inline) or request_uri (by reference, fetched over HTTP) is verified against the client's registered keys, and the parameters inside the JWT take precedence over the query string, per §6.1.

What is not accepted: alg: none (an unsigned request object destroys the exact integrity guarantee JAR exists for), a wrong-key signature, a mismatched inner client_id, an expired object, and — under Enforce — an algorithm that disagrees with the client's declared RequestObjectSigningAlg. All of them come back as invalid_request_object.

Rollout is staged: Off → LogOnly → Enforce. Discovery advertises request_parameter_supported, request_uri_parameter_supported, and the algorithm list only when JAR is on — a server shouldn't promise in metadata what the endpoint won't actually do. Configuration lives in the Jar section: enforcement mode, allowed algorithms (asymmetric only), clock skew, size limits, and the SSRF knobs for request_uri.

Now the part that was most of the work. OpenIddict 6.3.0 does not support request objects. We checked before planning: the assembly carries ValidateRequestParameter and the request_not_supported string, but nothing that parses a request object or verifies its signature, and request_uri exists only as a PAR URN. So JAR here isn't a config flag — it's our own server handlers (ValidateRequestObjectHandler) that slot in ahead of the built-in unconditional rejection and do the work themselves. PAR-issued urn:ietf:params:oauth:request_uri:* identifiers are deliberately left alone: PAR resolves those itself, and a JAR handler has no business there. request_uri fetches are size- and timeout-bounded and pass the SSRF filter below.

Coverage: 15 handler tests plus a live conformance demo, demo_jar_request_object.ps1, wired into the run_all sweep — so the scenario is exercised against a running server, not just a unit test.

What's not here yet: request-object encryption (JWE) is deferred — the client's RequestObjectEncryptionAlg / ...Enc are stored but not applied; that's a separate phase. RequestObjectSigningAlg, previously stored and ignored, is now genuinely enforced under Enforce, and JwksUri is genuinely resolved. The full six-phase plan, with risks (SSRF, alg:none, algorithm substitution) and sizing, is in the repo at doc/JAR_RFC9101_PLAN.md.

Resolving client keys

Underneath all this sits IClientKeyResolver — the resolver for the keys that verify what a client signed: a JAR request object today, a private_key_jwt assertion later. Two sources: the inline JsonWebKeySet on the application, or JwksUri, fetched and cached with background refresh on a TTL plus a rate-limited forced refresh on a kid miss, so a client rotating its keys doesn't break sign-in until the TTL expires.

Two principles, both about not letting uncertainty turn into permission:

Fails closed. An unreachable or malformed JWKS yields no keys, so there's nothing to verify against and the caller rejects. "Couldn't check" is never treated as "valid."
A broken inline JWKS never falls back to jwks_uri. A typo in a pasted key set has to stay visible instead of being papered over by a silent switch to another key source.

Asymmetric algorithms only: ClientSecret is stored as a BCrypt hash and verifying an HMAC needs the original secret, so HS* is impossible here on mechanics alone — and FAPI 2.0 forbids it for request objects regardless. Resolver settings live in the ClientKeys section: cache lifetime, minimum refresh interval, fetch timeout, maximum document size, plus RequireHttps and AllowPrivateNetworkTargets — two development-only relaxations that have no business in production.

The SSRF guard

Anything arriving from outside as a URL (jwks_uri, request_uri) goes through OutboundUrlGuard — before a socket is opened. It rejects anything that isn't an absolute HTTPS URL, and anything resolving to a non-public address: loopback, RFC 1918, link-local — including 169.254.169.254, the cloud metadata endpoint that hands instance credentials to whatever can reach it — CGNAT, IPv6 unique-local, and the IPv4-mapped forms like ::ffff:10.0.0.1 that a naive check walks straight past.

40 tests, 29 of them on the guard alone — including one asserting that 172.32.* and 172.15.* are public and must not be blocked (the 172.16/12 boundary is a classic off-by-one). All of them stay offline: attempting a network call fails the test by itself. Stating the known limitation myself: this does not close DNS rebinding, which needs the validated address pinned onto the connection itself.

About conformance — why SKIPPED is the right answer

After enabling JAR, the local OpenID Foundation Basic OP run went from 1 SKIPPED / 4 REVIEW to 2 SKIPPED / 3 REVIEW, with 0 FAILED throughout.

That deserves an explanation, because the number reads worse than the reality. Both SKIPPED modules test the unsigned (alg:none) request object. The suite skips them when the server doesn't advertise none among its supported algorithms — which is precisely our position: we support signed request objects only. Better still, one module moved REVIEW → SKIPPED because the server now genuinely processes signed request objects and advertises that honestly, instead of the previous no-support state that sent the test down a screenshot path.

There's exactly one way to turn either skip into a pass: start accepting alg:none. We're not trading a security property for a nicer line in a report. The full write-up is in OPENID_CERTIFICATION.md §4.3, so anyone looking at the badge sees the same picture we do.

Part 7. Connector odds and ends

Briefly, so it doesn't get lost. The Telegram connector (which got its own post) picked up some ergonomics:

replyToMessageId as a first-class producer option, expression-capable, with ReplyTo(long) / ReplyTo(IExpression) / .ReplyToIncoming(). Replying to the message that triggered the exchange no longer needs a hand-written .Process step copying one header into another.
messageId for edit/delete — same story: send-then-edit is now Tg.Edit(token).MessageId(Header(TelegramHeaders.SentMessageId)).
showAlert in answer mode — the callback answer shows as a modal alert instead of a toast.
A telegram.caption header for document/photo that wins over the option, consistent with how parseMode and fileName already behaved.
Mini App payloads (WebApp.sendData) now reach the route: the data becomes the exchange body — same contract as text messages and callback queries — and is also exposed as headers. Works on both the long-polling and webhook paths (they share a mapper); filter on telegram.messageType = "WebAppData".
And a fix: document / photo modes now genuinely pass telegram.replyToMessageId and telegram.replyMarkup through, so a photo with inline buttons keeps its keyboard.

Upgrading from 3.3.x

The release is backward compatible and existing routes are unchanged. Three things are worth a look, though.

The SQLite native extension was renamed: redb → redbsqlite. The package now ships runtimes/<rid>/native/redbsqlite.{dll,so} (win-x64, linux-x64, linux-arm64). The reason is mundane: the generic name collided with the managed redb.* assemblies and, worse, matched the redb.* prune globs a host applies to its own bin — a native loadable module was getting swept up as if it were one of ours. The C init symbol is unchanged (sqlite3_redb_init) and the binaries are byte-for-byte the ones shipped in 3.3.3 — this is a file rename, not a rebuild.
- Nothing to do if the package resolves the path for you (SqliteDataSource.LocatePackagedExtension(), the default for the Free DI registration) — it looks for the new name.
- Action required if you pin the path yourself: REDB_SQLITE_EXTENSION, an explicit NativeExtensionPath, a Dockerfile COPY, or a deploy script that copies redb.so by name. A stale path fails at connection open, not at build — i.e. at runtime.
Tsak: roles are now enforced. Roleless keys keep full access by default (with a one-time warning), so nothing should break on day one. Reissue keys with explicit roles, then set Tsak:Auth:RolelessKeysAreAdmin=false. Check that your CI scripts use a key with the right level: route force-stop and module deletion are now admin, diagnostics and logs are operator.
Tsak: the distribution build changed. The framework is now built into the shared layer: scripts/build-shared.ps1 -IncludeFramework (publish does this for you via publish/build.ps1). Running the Worker without it hits the fail-fast, and the message tells you exactly what to run. Prebuilt 3.4.0 images and archives are already correct.
Implementing IModuleHealthContributor? It moved from redb.Tsak.Core to redb.Tsak.Contracts: change using redb.Tsak.Core.Contracts; to using redb.Tsak.Contracts;. Nice side effect — your module can now reference only the lightweight contracts assembly and drop the redb.Tsak.Core reference entirely.

Getting it

Core and providers from NuGet:

dotnet add package redb.Core
dotnet add package redb.Postgres      # or redb.MSSql / redb.SQLite
dotnet add package redb.Postgres.Pro  # Pro — free, no key

The engine and whichever connectors you need:

dotnet add package redb.Route
dotnet add package redb.Route.Telegram
dotnet add package redb.Route.Kafka

The runtime as an image or an archive:

docker pull ghcr.io/redbase-app/redb-tsak-stack:3.4.0
# or the standalone archive from the v3.4.0 GitHub release (linux-x64 / win-x64), cosign-signed

Images are cosign-signed; the public key ships with the release (cosign.pub):

cosign verify --key cosign.pub ghcr.io/redbase-app/redb-tsak-worker:3.4.0

Everything targets .NET 9.

Wrapping up

If 3.3.0 was about the stack holding up under load, 3.4.0 is about the stack being operable.

What failed gets replayed from a save-point instead of from whatever the state decayed into on the way down — and it's a button in a dashboard, not a night spent reading logs. A library patch ships as a DLL swap, with no runtime rebuild and no re-cut archives. A node leaves the rotation gracefully instead of being evicted. Secrets stay out of logs, telemetry, and the dashboard — and better yet, never enter the route. The control plane hands out permissions by role, module code is verified against a signature before it loads, and the record of who did what survives a restart. A scheme's identity in the database stopped being hostage to your namespace. And the whole ecosystem, Identity included, finally rides one version number.

This is the class of capability nobody sees in a demo and everybody feels in production.

If you take it for a spin, tell me what landed and what's missing — deep dives on replay checkpoints and on the secret-redaction work are both queued up, and I'd rather aim them at real questions.

Sources and releases: github.com/redbase-app. About the database: redbase.app. Previous posts: my dev.to profile.

If this was useful — a ⭐ on GitHub helps others find it.

Basic OP: 35 modules, 0 failures. Config OP: 0 failures. Found and fixed: PII leak in id_token, missing Cache-Control: no-store. report: https://github.com/redbase-app/redb-identity/blob/main/OPENID_CERTIFICATION.md #dotnet #oidc #oauth

rinat kozin — Wed, 22 Jul 2026 20:05:11 +0000

redb-identity/OPENID_CERTIFICATION.md at main · redbase-app/redb-identity · GitHub

Transport-agnostic OAuth 2.1 / OpenID Connect server for the redb ecosystem — OpenIddict on redb.Route, redb-backed stores, MFA, WebAuthn, federation, SCIM, DPoP. Green on PostgreSQL, MSSQL and SQL...

github.com

The night job that carves its own partitions: cron, a plpgsql function, and a Splitter

rinat kozin — Tue, 21 Jul 2026 17:17:47 +0000

Series: redb ecosystem

There's a table of GPS points. Vehicles push coordinates every few seconds, a day adds up to millions of rows, and the table is partitioned by time. Which means somebody has to create tomorrow's partition ahead of time and detach the ones older than ninety days. Otherwise, one fine night, an insert dies with no partition of relation "gps_points" found for row — at precisely the hour nobody is watching.

The problem is as old as the hills, and it's usually solved one of two ways: pg_cron inside the database, or an IHostedService with a PeriodicTimer in the app. The first has an observability problem — the job lives in the database, and your application logs know nothing about it. The second has the problem that a small private infrastructure grows around PeriodicTimer very quickly: retries, logging, "what if the previous run is still going", "what if there are three nodes".

I'll show a third way — a route. What follows is a walkthrough of the redb.Route.Quartz connector, calling a PostgreSQL function through the SQL connector, a splitter with error isolation, and an honest conversation about clustering, because that's where it actually gets interesting.

All the code in this post uses string URIs. redb.Route has fluent builders too, but a URI reads without knowing the API, and you can paste it into a config file.

redb ecosystem series. This is a continuation — newest posts first:

The Amazon SQS + SNS connector in redb.Route, at-least-once and pub/sub via SNS→SQS. Still leaving MassTransit

The RabbitMQ connector in redb.Route: the whole broker in one URI — RPC, competing consumers, dead-lettering

redb 3.3.0: an enterprise .NET stack you actually own — typed store, a homegrown Apache Camel, and a runtime with a dashboard (all free)

Two routes in an evening: from a debug worker to an enterprise runtime with redb.Route + Tsak

Leaving MassTransit for a Camel state of mind: the Kafka connector, Scatter-Gather, and what really happens under a transaction

Full list on the profile. Sources: github.com/redbase-app/redb-route. About the database itself: redbase.app.

The SQL first

There's no magic at this level, so let's start from the most honest place — the function in the database. It takes a table name and a retention window, creates tomorrow's partition if it isn't there yet, and detaches everything past the window:

CREATE OR REPLACE FUNCTION maintain_partitions(tbl text, keep_days int)
RETURNS text AS $$
DECLARE
    next_day   date := (now() + interval '1 day')::date;
    part_name  text := format('%s_%s', tbl, to_char(next_day, 'YYYYMMDD'));
    cutoff     date := (now() - make_interval(days => keep_days))::date;
    dropped    int  := 0;
    old_part   text;
BEGIN
    IF to_regclass(part_name) IS NULL THEN
        EXECUTE format(
            'CREATE TABLE %I PARTITION OF %I FOR VALUES FROM (%L) TO (%L)',
            part_name, tbl, next_day, next_day + 1);
    END IF;

    FOR old_part IN
        SELECT c.relname
        FROM pg_class c
        JOIN pg_inherits i ON i.inhrelid = c.oid
        JOIN pg_class p ON p.oid = i.inhparent
        WHERE p.relname = tbl
          AND c.relname < format('%s_%s', tbl, to_char(cutoff, 'YYYYMMDD'))
    LOOP
        EXECUTE format('DROP TABLE %I', old_part);
        dropped := dropped + 1;
    END LOOP;

    RETURN format('%s: +1 partition, -%s dropped', tbl, dropped);
END;
$$ LANGUAGE plpgsql;

The function returns a string so it shows up in the log. That "so it shows up in the log" is the single concession to convenience here; everything else is ordinary plpgsql you'd have written anyway.

The tick: `cron://`

The redb.Route.Quartz connector gives you two schemes. The first is cron:, a regular Quartz trigger with a cron expression:

cron://[group/]jobName?schedule=<cron-expression>&<options>

The second is qtimer:, a plain periodic trigger for when you don't need cron:

qtimer://[group/]jobName?period=5000&delay=1000&fixedRate=true

There is deliberately no quartz: scheme. That's not an oversight: scheduler setup — where jobs are stored, whether it's a cluster or a single node, what the thread pool looks like — is a property of the host, not of the route. Only the schedule belongs in a route URI. If a quartz: scheme existed, job store settings would immediately start growing into it, and the route would stop being portable.

The cron expression is the Quartz one, six fields, with seconds. Let's put partition maintenance at 02:30:

From("cron://maintenance/gps-partitions?schedule=0 30 2 * * ?")
    .RouteId("gps-partitions")

The expression is validated when the endpoint is created, not when it first fires. A typo in the schedule is an ArgumentException at application startup, not silence until three in the morning.

Registering the component is one line at the module entry point:

context.AddComponent(new CronComponent());

Calling the function: `sql:`

The SQL connector has a single scheme, sql:, and the mode is picked with the mode parameter. To call a PostgreSQL function there's mode=Procedure with the asFunction=true flag — the connector then assembles SELECT maintain_partitions(@p1, @p2), executes it as a scalar, and puts the result in the message body:

sql:maintain_partitions
  ?mode=Procedure
  &dataSource=#pg
  &procedureName=maintain_partitions
  &asFunction=true
  &procedureParams=IN:tbl:String,IN:keep_days:Int32
  &param.keep_days=90

procedureParams declares the parameters in direction:name:type form, and the declaration order is the argument order in the call. There are three directions — IN, OUT, INOUT — and OUT values come back into the message headers under their own names after execution.

A parameter's value is resolved along a chain: first an explicit param.name from the URI, then a message header with the same name, then the body if it's a dictionary. Here keep_days is a constant right in the URI, and tbl will arrive from a header set by the splitter.

For simple cases there's a shorter path — mode=Execute (the default) with @name placeholders directly in the query text:

sql:SELECT maintain_partitions(@tbl, @keep_days)
  ?dataSource=#pg
  &outputType=Scalar
  &param.tbl=${header.tbl}
  &param.keep_days=90

Note ${header.tbl} — that's an expression, resolved at runtime from the message header. Placeholders in SQL are @name only, the colon form isn't supported, and an unsubstituted parameter silently becomes NULL, so it pays not to mistype the names.

Both variants work. For the rest of the post I use mode=Procedure because it's the more illustrative one.

The splitter: three tables, and one failure doesn't take down the rest

There's rarely just one time-partitioned table. We have three: points, tracks, and events. The naive move would be a loop inside a processor, but then you're deciding by hand what happens when the second table fails — abort everything or carry on, and how do you find out afterwards what didn't run.

This is exactly the EIP Splitter pattern. A message carrying a list is split into one message per element, each goes down its own branch, and the branches can be processed in parallel:

From("cron://maintenance/gps-partitions?schedule=0 30 2 * * ?")
    .RouteId("gps-partitions")
    .Process(e => e.In.Body = new[] { "gps_points", "gps_tracks", "gps_events" })
    .Split(Body())
        .ParallelProcessing()
        .MaxParallelism(2)
        .SetHeader("tbl", Body())
        .DoTry()
            .To("sql:maintain_partitions"
                + "?mode=Procedure"
                + "&dataSource=#pg"
                + "&procedureName=maintain_partitions"
                + "&asFunction=true"
                + "&procedureParams=IN:tbl:String,IN:keep_days:Int32"
                + "&param.keep_days=90")
            .Log("[PART] ${body}")
        .DoCatch<Exception>()
            .Log("[PART] ${header.tbl}: ${exception.Message}", LogLevel.Error)
        .End()
    .EndSplit()
    .Process(Summary);

Twelve lines, and they already contain everything you'd normally bolt on a week later. MaxParallelism(2) means two tables get serviced at once and the third waits for a free slot; DROP TABLE takes an ACCESS EXCLUSIVE lock, and there's no point burying the database under parallel locks. DoTry/DoCatch sit inside the split, so the exception is isolated to its own branch: a failing gps_tracks won't undo the gps_points that already succeeded, and won't stop gps_events. After EndSplit, control lands in Summary, where you can count how many branches made it and decide whether to page somebody.

This isn't a trick invented for the article. Exactly this shape runs in production — a job that syncs shipping points from SAP splits the list of points and processes three at a time, because one unreachable point shouldn't take down the whole tick:

From("timer://tsum-points?period=180000&delay=60000")
    .RouteId("tsum-points-timer")
    .ProcessWithRedb(PreloadContextAsync)
    .Split(Body())
        .ParallelProcessing()
        .MaxParallelism(3)
        .SetHeader("ShippingPoint", Body())
        .DoTry()
            .To(sqlTo)
            .Process(DeserializeXml)
            .ProcessWithRedb(ProcessPointsAsync)
        .DoCatch<Exception>()
            .Process(AddPointSyncError)
            .Log(LogLevel.Error)
                .Message("[TSUM-PT] SP=${header.ShippingPoint} failed, skipping: ${exception.Message}")
            .EndLog()
        .End()
    .EndSplit()
    .Process(BuildPointSyncSummary);

The splitter isn't the only EIP that docks onto a scheduler. A schedule pairs naturally with a Content-Based Router (one thing on weekdays, another on weekends), a Throttler (don't hammer an external API more than N times a second), an Aggregator (collect branch results into a single report), an Idempotent Consumer (more on that below) and a Dead Letter Channel. redb.Route implements nearly the whole Hohpe & Woolf catalogue — Splitter, Aggregator, Resequencer, Multicast, Recipient List, Dynamic Router, Wire Tap, Content Enricher, Claim Check, Saga, Scatter-Gather, Circuit Breaker, Load Balancer, Transactional Client and the rest. The scheduler here is just a source, not a separate world with its own rules.

What happens when the server was down at 02:30

This is where the reason you'd want Quartz rather than a PeriodicTimer starts.

The job didn't fire, because the node was down or a deploy ran long. What should happen when the scheduler comes back at 02:47? The answer depends on the job, and it isn't a philosophical question: for partition maintenance, a skip is a catastrophe — tomorrow's partition has to be created, at 02:47 or at 06:00. For a "send the morning report" job, running at noon is worse than not running at all.

Quartz calls this misfire, and the connector passes the policy straight through the URI:

cron://maintenance/gps-partitions?schedule=0 30 2 * * ?&misfireInstruction=CronFireOnceNow

CronFireOnceNow — catch up, run once, return to the schedule. CronDoNothing — skip it, wait for the next scheduled time. Simple triggers (qtimer:) have more policies — five of them — differing in what to do with the accumulated repeat count. But the choice always collapses to one question: does a missed run need catching up, or has it already gone stale?

The connector's default for qtimer: is picked by common sense: with fixedRate=true, catch up (you asked for a fixed rate); otherwise, reschedule from the next tick.

What happens when the previous run is still going

Partitions have piled up, DROP TABLE is waiting on a lock, the job is hanging. The next fire time arrives. Now what?

The standard Quartz answer is the [DisallowConcurrentExecution] attribute on the job class. The connector went a different way: concurrency is governed by the consumer's own semaphore, and if every thread is busy the fire is silently skipped:

// QuartzConsumerBase.cs
if (!await _semaphore.WaitAsync(0).ConfigureAwait(false))
    return; // all threads busy, skip this fire

The semaphore size is set in the URI via threads (default 1). Why not the attribute: [DisallowConcurrentExecution] is either one run or no control at all, with nothing in between. A semaphore lets you say "up to three concurrent runs of this job" while staying friendly to a cluster, where the limit lives at the job store level rather than in a class attribute.

If you do want the Quartz semantics, there's a stateful=true flag that switches the job to a class carrying [DisallowConcurrentExecution] and [PersistJobDataAfterExecution].

One more detail about shutdown: when a route is stopped, the connector unschedules the trigger and waits for runs already in flight — up to thirty seconds. A job that happens to be dropping a partition right then won't be cut off halfway. And if a job does fire when its route is already gone (the module was unloaded, say), the job notices on startup that its consumer is dead and deletes itself from the scheduler, leaving no litter behind.

Three nodes

The most common question about cron in a distributed application: if there are three nodes, does the job fire three times?

It does — if every node keeps its own scheduler in memory. That's exactly how the connector's fallback works: no IScheduler found in the context, so it creates its own, RAM-backed, unique to that context. Fine for local development, not fine for three nodes.

The right answer is one scheduler per cluster — more precisely, one shared job store. Quartz does this out of the box via AdoJobStore, and the connector deliberately stays out of the way: it does not create its own scheduler if a ready one is already sitting in the route context. The host drops a clustered one in there, and the route needs no changes at all — the URI stays the same.

And here's where all the plumbing that usually gets hand-waved away comes into view. AdoJobStore is a set of QRTZ_* tables in your database. QRTZ_TRIGGERS holds the next fire time, QRTZ_FIRED_TRIGGERS holds who is executing what right now, QRTZ_LOCKS holds row-mutexes. The "only one node runs the job" mechanism isn't clever consensus — it's SELECT ... FOR UPDATE on a row in QRTZ_LOCKS: whoever takes the lock first takes the trigger. Each node checks in periodically in QRTZ_SCHEDULER_STATE, and if a node stops checking in, another one picks up its unfinished jobs — but only those marked recoverableJob=true. The flag is off by default: restarting a job when you don't know whether it's idempotent is a bad idea.

The table schema is created by the host at startup; the connection string and dialect come from the application's database configuration. So you don't write the DDL, but the tables are perfectly ordinary, they sit next to yours, they're visible in any client, and when something goes wrong you go in with a plain SELECT and see which trigger is stuck and on which node.

And since we brought up recoverableJob: recovery after a node dies means the job may run twice. For our partition maintenance function that's safe — it's written idempotently (IF to_regclass(...) IS NULL), and that's a requirement, not an accident. If the job weren't idempotent — say, it credited bonuses — you'd put an Idempotent Consumer in front of it, and redb.Route has one backed by SQL with a unique index. A unique index, not a "smart cache": in a cluster, that's the only thing that saves you from double execution.

What lands in the message

A scheduler is a source with no message body. The body is null, the pattern is InOnly, and everything about the firing lives in the properties:

Property	What's in it
`CamelQuartzFireTime`	when the job actually fired
`CamelQuartzScheduledFireTime`	when it should have fired per the schedule
`CamelQuartzNextFireTime`	when it fires next
`CamelQuartzPreviousFireTime`	when it fired last
`CamelCronSchedule` / `CamelCronName` / `CamelCronGroup`	the expression, name and group of the job

The gap between FireTime and ScheduledFireTime is that misfire, in numbers. If they're seventeen minutes apart, the job was catching up.

The Camel-style names aren't nostalgia. redb.Route deliberately keeps Apache Camel's nomenclature wherever the semantics match, so someone arriving from Java integrations can read the headers without a dictionary.

Two jobs from production

So this doesn't read like a piece about a spherical cron in a vacuum — here are two routes that run every night in a transport management system.

Database backup at three:

From("cron://tsum-backup?schedule=0 0 3 * * ?")
    .RouteId("tsum-backup-cron")
    .ProcessWithRedb(RunBackupAsync);

Dead-route cleanup at four:

From("cron://tsum-cleanup?schedule=0 0 4 * * ?")
    .RouteId("tsum-cleanup-cron")
    .ProcessWithRedb(CleanupDeadRoutesAsync);

Nothing spectacular, and that's the point: the schedule is in the URI, the logic is in the processor, retries and logging come from the framework. Note that the schedule is hardcoded in the URI rather than pulled from config — that's allowed, and it's how things live for a while. When you need to change a schedule without a rebuild, you assemble the URI from config with ordinary concatenation, because it's just a string.

The full option list

So this doesn't turn into half a post of reference material — everything cron: understands:

schedule (required), timeZone (IANA name), threads, misfireInstruction, stateful, recoverableJob, durableJob, deleteJob, pauseJob, startAt, endAt, customCalendar, triggerStartDelay, prefixJobNameWithEndpointId.

qtimer: swaps schedule for period, delay, fixedRate, repeatCount; everything else is the same, minus the time zone (a simple trigger doesn't need one).

customCalendar is worth calling out: it's the Quartz exclusion calendar, registered in the context by name. That's how you get "except holidays" and "business days only" — things a cron expression cannot express at all.

Wrapping up

The scheduler in redb.Route is a message source, not a subsystem with its own rules. A job is a route, which means everything available to any route is available to it: the splitter, exception handling, transactions, retries, metrics. The trigger is described by a single URI string, and there's nothing about infrastructure in that string — only the schedule.

The infrastructure isn't hidden, either. The cluster runs on QRTZ_* tables and a row lock in the database, idempotency in a cluster comes from a unique index, and partition maintenance is an ordinary plpgsql function that you wrote and can read. The framework here spares you the glue code, not the understanding of what's happening in your database.

Code: github.com/redbase-app/redb-route · Docs: redbase.app

If this was useful — a ⭐ on GitHub helps others find it.

The Amazon SQS + SNS connector in redb.Route, at-least-once and pub/sub via SNS SQS. Still leaving MassTransit

rinat kozin — Thu, 16 Jul 2026 13:24:18 +0000

series: "redb.Route"

We've done Kafka and RabbitMQ earlier in this series. Next up: Amazon SQS — and riding along in the same package, SNS. Two transports, because in AWS-land they travel together. The redb.Route.Sqs connector sits on the native AWS SDK for .NET v4 (AWSSDK.SQS, AWSSDK.SimpleNotificationService), but you don't write a "client" — you write routes, and the whole queue collapses into a single URI:

From("sqs://orders?waitTimeSeconds=20&concurrentConsumers=4")
    .Log("Order in: ${body}")
    .To("direct://process");

Read it once and the whole story's there: long-poll the orders queue in 20-second pulls, four consumers wide, log it, pass it on. No "new up a client," no ReceiveMessage, no "remember to DeleteMessage after you've handled it" — the connector does all of that.

About "leaving MassTransit," same disclaimer as last time: it's not that MassTransit can't do this. It's had SQS/SNS for years — MassTransit.AmazonSQS — and it wires up the same SNS→SQS pairing. The difference is the model. MassTransit is a bus — message contracts, consumers, bus config, the whole worldview bought at once. redb.Route is explicit routes in the Apache Camel spirit: an endpoint is a URI, the integration patterns are steps in a route, the transport is an abstraction underneath. This post is about what SQS and SNS look like in that second model.

And the thing everyone's actually asking about right now — worth stating precisely rather than gleefully. MassTransit went commercial with v9: the line now ships under Massient, Inc., and MassTransit.AmazonSQS 9.0.0 on NuGet is already from there. But no spin: v8 stays Apache 2.0, with security patches and critical fixes through at least the end of 2026; v9 isn't a closed binary — it's source-available; and organizations under $1M USD annual revenue (plus non-profits under $1M in expenses) get it at a 100% discount — free, minus commercial support. So "MassTransit is paid now" is false for a small team and true for a large one, where it turns into a real decision: pay, ride v8 to the end of its support, or look around. redb.Route is Apache 2.0, this SQS connector included — but pick it because the routing model fits your head, not because of a license line. A license is a reason to look around; it isn't an argument about architecture.

All the code is in English; the examples aren't invented — they run against LocalStack (a Docker container on http://localhost:4566), the same way the connector's integration tests do.

Part of the redb / redb.Route series — recent posts first:

The RabbitMQ connector — the whole broker in one URI: RPC, competing consumers, dead-lettering (previous in the series)

Leaving MassTransit for a Camel state of mind: the Kafka connector, Scatter-Gather, and transactions

Apache Camel for .NET, dissected: the HTTP connector with no ASP.NET MVC + the Content-Based Router

redb.Route — Apache Camel for .NET: 22 transports, 30+ EIP patterns, compiled DSL

Sources: github.com/redbase-app. About the database itself: redbase.app.

One package, two transports

redb.Route.Sqs registers two schemes, and they don't play the same role:

sqs:// — a queue. Works as both a consumer (in From(...)) and a producer (in .To(...)).
sns:// — a topic. Publisher-only. SNS has no pull consumer — delivery is push (to SQS / HTTP / email / SMS). Try to From("sns://...") and it throws NotSupportedException on purpose: to "read" a topic, you subscribe an SQS queue to it and read that with sqs://.

Why they share a package: at AWS they're a pair. SQS is a durable point-to-point queue (one message, one consumer). SNS is a publish-subscribe topic (one message, every subscriber). The canonical AWS fan-out is an SNS topic with several SQS queues hanging off it — so one connector covers both: sqs:// and sns://.

Wiring it into DI:

services.AddRedbRoute(route =>
{
    route.Services.AddRedbRouteSqs();   // registers both sqs:// and sns://
    route.AddRouteBuilder<MyRoutes>();
});

Anatomy of the URI

An endpoint is a string:

sqs://queue-name?region=us-east-1&waitTimeSeconds=20&concurrentConsumers=4
sns://topic-name?region=us-east-1&autoCreateTopic=true

Scheme, the queue or topic name in the path, then query parameters (names are in the tables below). The fluent builder gives you the same thing: Sqs.Queue("orders").WaitTimeSeconds(20)... and Sns.Topic("events").Region("us-east-1")... build that exact string, URL-encoding included.

Three things are AWS-specific and worth a pause:

Don't repeat your credentials. Twenty queues on one account? Register an AwsConnectionFactory in the registry and point at it with connectionFactory=name. One factory builds both the SQS and the SNS client — they share credentials, region and service URL:

context.AddToRegistry("prod-aws", new AwsConnectionFactory
{
    Region = "eu-west-1",
    UseDefaultCredentialsProvider = true,   // IAM role on the instance / SSO / env vars
});

From("sqs://orders?connectionFactory=prod-aws&concurrentConsumers=4");
To("sns://events?connectionFactory=prod-aws");

LocalStack / ElasticMQ via serviceUrl=. An explicit URL overrides the regional endpoint (the region sticks around only to sign the request). That's your local-dev and test mode:

sqs://orders?serviceUrl=http://localhost:4566&region=us-east-1&accessKey=test&secretKey=test

Queues and topics get created on the spot. autoCreateQueue=true (or autoCreateTopic=true) creates the resource on startup if it's missing. A name ending in .fifo comes up as a FIFO resource automatically, with ContentBasedDeduplication=true.

Every parameter

The reason to bookmark this post. Names are exactly as they appear in the URI; defaults are what you get out of the box.

Connection and credentials (shared by `sqs://` and `sns://`)

Parameter	Default	What it's for
`region`	`us-east-1`	AWS region
`serviceUrl`	(empty)	Explicit endpoint (LocalStack/ElasticMQ); overrides the region
`accessKey` / `secretKey`	(empty)	Static keys
`sessionToken`	(empty)	Temporary (STS) credentials
`profileName`	(empty)	A named profile from the shared credentials file
`useDefaultCredentialsProvider`	`false`	The default AWS chain (env / IAM role / SSO)
`connectionFactory`	(empty)	A factory in the registry; overrides URI credentials
`retryCount`	`3`	SDK retries on transient errors
`connectionTimeout`	`30000`	HTTP client timeout, ms
`proxyHost` / `proxyPort`	—	HTTP proxy

Resolution order: the default chain → a profile → a session token → static keys. Validate() insists on at least one path — accessKey+secretKey, profileName, useDefaultCredentialsProvider=true, or a registered connectionFactory — otherwise it throws at startup.

SQS — consumer (`From("sqs://...")`)

Parameter	Default	What it's for
`waitTimeSeconds`	`20`	Long-poll (0–20). Fewer empty receives, fewer API calls
`maxNumberOfMessages`	`10`	How many to pull per receive (1–10)
`visibilityTimeout`	`0` (queue default)	How long a message stays hidden while you work it
`concurrentConsumers`	`1`	Number of competing receive loops (see the concurrency recipe)
`extendMessageVisibility`	`false`	Keep extending visibility while the handler runs (a heartbeat); needs `visibilityTimeout > 0`
`deleteAfterRead`	`true`	Delete the message after a clean pass (this is the ack)
`resetVisibilityOnFailure`	`false`	On error, reset visibility to 0 → immediate redelivery
`transacted`	`false`	Defer the delete into the route transaction (`.Transacted()`)
`attributeNames`	`All`	Which system attributes to request
`messageAttributeNames`	`All`	Which message attributes to request
`delay`	`0`	Pause (ms) after an empty receive before polling again
`initialDelay`	`0`	Delay (ms) before the first poll

SQS — producer (`.To("sqs://...")`)

Parameter	Default	What it's for
`delaySeconds`	`0`	Delivery delay (0–900). FIFO queues reject a per-message delay
`messageGroupId`	—	FIFO group id. A constant or a `${...}` expression (per message)
`messageDeduplicationId`	—	FIFO dedup id. Skip it if the queue has content-based dedup
`enableBatch`	`false`	Send an `IEnumerable` body as one `SendMessageBatch`
`batchMaxMessages`	`10`	Batch size (SQS's hard cap is 10)
`autoCreateQueue`	`false`	Create the queue on startup (FIFO if the name ends in `.fifo`)
`queueUrl`	(empty)	An explicit queue URL instead of resolving by name

SNS — publisher (`.To("sns://...")`)

Parameter	Default	What it's for
`autoCreateTopic`	`false`	Create the topic on startup (FIFO on `.fifo`)
`topicArn`	(empty)	An explicit topic ARN instead of resolving by name
`subject`	(empty)	Subject (for email delivery). Supports `${...}`
`messageStructure`	(empty)	`json` — per-protocol payloads
`messageGroupId` / `messageDeduplicationId`	—	For FIFO topics
`subscribeSnsToSqs` + `subscribeQueueArn`	`false`	On startup, subscribe an SQS queue (by ARN) to this topic
`rawMessageDelivery`	`false`	On that subscription, deliver the bare payload (see the pub/sub recipe)

Headers and tracing through the broker

The sqs:// consumer stamps message metadata onto the headers (prefixed redbSqs.), and incoming message attributes land under redbSqs.attr.:

Header	What's in it
`redbSqs.queue`	The queue name
`redbSqs.messageId`	Message id
`redbSqs.receiptHandle`	Receipt handle (needed to delete or change visibility)
`redbSqs.approximateReceiveCount`	How many times this message has been delivered ("is this a retry?")
`redbSqs.messageGroupId` / `redbSqs.sequenceNumber`	FIFO group + sequence number
`redbSqs.sentTimestamp`	When it was sent (epoch millis)
`redbSqs.attr.<name>`	An incoming user message attribute

Going the other way, the header → message-attribute mapping on the producer goes like this: user headers ride out as attributes as-is; an incoming redbSqs.attr.<name> is forwarded under the bare <name> (the prefix is stripped, so sqs→sqs and sqs→sns bridges keep their attributes); the internal redbSqs.* / redbSns.* keys are dropped, so one hop's metadata doesn't leak into the next.

Tracing comes for free: before sending, the producer injects the W3C traceparent/tracestate as message attributes via the standard DistributedContextPropagator, and the consumer on the far end picks them up and continues the trace. Over plain SQS (sqs:// → sqs://) the chain is unbroken with zero setup. (There's a wrinkle when SNS is in the middle — see the pub/sub recipe.)

Straight talk on delivery: at-least-once, visibility, transacted

This is where the marketing likes to say "exactly-once." Let's read the code instead.

The SQS consumer is at-least-once. A message is deleted (DeleteMessage) only after it clears the route. The settle logic:

clean pass + deleteAfterRead=true → DeleteMessage (the ack);
failed + resetVisibilityOnFailure=true → ChangeMessageVisibility(0) → it comes right back (fast retry);
failed without that flag → it simply times out on visibility and gets redelivered.

visibilityTimeout is how long the message is hidden while you work it. Long handler? There's extendMessageVisibility=true: a background heartbeat bumps the visibility every visibilityTimeout / 2, so slow work never triggers a redelivery out from under you.

transacted=true rolls the ack into the route transaction. The delete doesn't happen immediately — a deferred SqsAckAction is registered, whose Commit deletes the message and whose Rollback resets visibility to 0 (immediate redelivery). Commit and rollback ride along with your redb work at the .Transacted() boundary:

From("sqs://orders?transacted=true&visibilityTimeout=60")
    .Transacted()
        .ProcessWithRedb((redb, ex) => redb.SaveAsync(Parse(ex)))   // write to the DB
    .EndTransaction();     // DB committed → the message is deleted; DB failed → it comes back

What's not here is cross-queue "read from A, write to B" atomicity at the broker level — SQS doesn't have it to give. Crash in the gap between "work done" and "delete," and after a restart you get a repeat — at-least-once, not exactly-once. So make your handler idempotent; redbSqs.approximateReceiveCount is there to help. The honest label is "at-least-once with a transactional ack at the route level" — no exactly-once fairy dust.

Ordering holds only on a FIFO queue (.fifo) and only with concurrentConsumers=1.

Recipe 1: producer, consumer, FIFO, batch

The bare minimum:

// Consumer — long-poll, delete after success
From(Sqs.Queue("orders").WaitTimeSeconds(20))
    .Process(HandleOrder);

// Producer — send the body to the queue
To(Sqs.Queue("orders").Region("eu-west-1"));

FIFO — the name ends in .fifo, and a group id is required (a constant or a per-message expression):

To(Sqs.Queue("orders.fifo").MessageGroupId("${header.customerId}"));

Batch — an IEnumerable body goes out as a single SendMessageBatch (in chunks of batchMaxMessages, SQS's cap being 10):

From("timer://tick?period=5000")
    .SetBody(_ => new[] { "a", "b", "c", "d", "e" })
    .To(Sqs.Queue("orders").EnableBatch());   // one call instead of five

Recipe 2: competing consumers (concurrency)

One parameter — concurrentConsumers:

From(Sqs.Queue("orders").ConcurrentConsumers(8).MaxNumberOfMessages(1))
    .Process(HandleOrder);   // up to 8 messages handled at once

This is the SQS-native concurrency model: the connector spins up N independent receive loops (one task each), and each pulls and handles its own message. Setting maxNumberOfMessages=1 alongside it makes each of the N workers hold exactly one message at a time, so the pool saturates honestly instead of one loop grabbing a batch of 10. The default is 1 (strictly serial). Go N > 1 and per-queue ordering is off the table, and your handler had better be thread-safe.

How much this actually parallelizes shows up in the SqsRpcDemo (more on it below): 12 requests, roughly 300 ms of work each, a pool of 4 — the whole batch clears in about 1.2 s instead of the ~3.6 s a serial consumer would spend, and the measured peak concurrency pins at exactly 4.

And if what you want to parallelize is the processing inside the route rather than the intake (a serial source, heavy work), there's an orthogonal EIP step, .Threads(N). concurrentConsumers scales reading off the queue; .Threads(N) scales the work; the CONCURRENCY.md guide walks through the difference. They compose freely.

Recipe 3: RPC (request/reply) — and why SQS won't hand it to you

Here's the honest contrast with the last post. On RabbitMQ, RPC flipped on with one flag — replyTo=true on the client and zero config on the server, because the broker does reply queues itself. SQS has none of that — no protocol-level reply-to, no built-in waiting for an answer. RPC over SQS is something you build, using the classic correlation pattern: a reply queue, a correlationId, and matching the answer back. That's exactly what the SqsRpcDemo shows.

The client tags each request with two message attributes — correlationId and replyTo (the queue to answer on) — and drops it on the request queue. The worker computes a result and sends it to the queue named in replyTo; the correlationId rides back on its own (the incoming redbSqs.attr.correlationId is forwarded straight back out as an attribute). The worker is a plain consumer plus a dynamic .ToD(...):

// Worker: N competing consumers on the request queue, reply to the queue named in replyTo
From(Sqs.Queue("rpc-requests").ConcurrentConsumers(4).MaxNumberOfMessages(1))
    .Process(async (ex, ct) =>
    {
        var n = int.Parse(ex.In.Body!.ToString()!);
        await Task.Delay(300, ct);              // simulated work
        ex.In.Body = (n * (long)n).ToString();  // the reply body
    })
    // dynamic destination — the reply queue comes from the request's replyTo attribute
    .ToD(ex => Sqs.Queue(ReplyTo(ex)).Build());

On the client side a separate route reads the reply queue and matches the correlationId back to the pending call:

From(Sqs.Queue("rpc-replies").MaxNumberOfMessages(10))
    .Process((ex, ct) =>
    {
        var id = Attr(ex, "correlationId");
        if (id is not null && Pending.TryRemove(id, out var tcs))
            tcs.TrySetResult(ex.In.Body?.ToString() ?? "");
        return Task.CompletedTask;
    });

The takeaway: RPC on SQS works, but it's assembled from primitives (attributes + .ToD + correlation), not toggled with a flag. When you want RPC out of the box, you reach for a broker with a native reply path (RabbitMQ). SQS is honest about what it is: durable queues and redelivery, and request/reply is a thing you wire up on top. The full, runnable version lives in redb.Route/demos/SqsRpcDemo (it runs against LocalStack).

Recipe 4 (EIP): Publish-Subscribe via SNS→SQS fan-out

Here's the headline pattern, the whole reason SQS keeps SNS company. Publish-Subscribe Channel, straight out of Hohpe & Woolf: the publisher sends one message and every subscriber gets it — independently, into its own queue, with its own retries. At AWS you don't do this in code, you do it with topology: an SNS topic with several SQS queues subscribed to it.

publish "order" ──▶ SNS topic ──┬──▶ SQS "orders-billing"  ──▶ billing route
                                 └──▶ SQS "orders-shipping" ──▶ shipping route

The publisher just publishes to the topic, and every subscribed queue gets a copy — each with its own independent consumer:

// Publisher — publish an order event; SNS fans it out to every subscribed queue:
From("timer://orders?period=5000")
    .SetBody(_ => BuildOrder())
    .To(Sns.Topic("orders-events"));

// Two independent subscribers — each its own SQS queue, each its own consumer:
From(Sqs.Queue("orders-billing")).Process(Charge);
From(Sqs.Queue("orders-shipping")).Process(Ship);

The subscriptions themselves the connector can set up from the URI — subscribeSnsToSqs + the queue ARN; the subscribe runs when the sns:// producer starts. Each sns:// endpoint subscribes one queue (a single ARN), so a two-queue fan-out is two subscriptions on the one topic:

// Subscribe each queue to the topic with raw delivery (bare payload, no envelope):
Sns.Topic("orders-events").SubscribeSnsToSqs(billingArn).RawMessageDelivery();
Sns.Topic("orders-events").SubscribeSnsToSqs(shippingArn).RawMessageDelivery();

For the exact fan-out wiring (creating the queues, reading their ARNs, subscribing, publishing), see redb.Route/demos/SqsPubSubDemo.

⚠️ The catch: the envelope vs. raw

By default SNS does not drop your payload into the queue as-is — it wraps it in a JSON notification envelope:

{ "Type": "Notification", "MessageId": "...", "TopicArn": "...",
  "Message": "<your payload, as a string>", "MessageAttributes": { ... } }

So the subscribing queue receives the envelope, not your message, and your SNS attributes sit inside it rather than as SQS attributes. That's standard AWS SNS behavior (the subscription's Raw Message Delivery = OFF), and it earns its keep in exactly one case: when a single queue listens to several topics and you need the envelope's TopicArn to tell them apart. For an ordinary fan-out it's just in the way — you end up unwrapping the envelope, and the SNS→SQS trace breaks (the traceparent goes inside the envelope, while the SQS consumer looks for it among the SQS attributes).

The fix is the subscription's Raw Message Delivery = true: the queue gets the bare payload, and the SNS attributes become SQS attributes (and the trace is whole again). In the connector that's .RawMessageDelivery() — it sets RawMessageDelivery=true on that subscribeSnsToSqs subscription. The SqsPubSubDemo prints the first delivered body: with .RawMessageDelivery() it's {"orderId":1,"amount":100}, not an envelope.

On rawMessageDelivery. This option lands in the next NuGet release of redb.Route.Sqs. You can already grab it from source right now — the redb.Route.Sqs connector on GitHub (the connector folder — SnsEndpointOptions / SnsProducer / the RawMessageDelivery fluent method). Without it, SNS→SQS still works; you just get the JSON envelope and unwrap it yourself (envelope.Message).

Recipe 5 (a cousin): Multicast — the same fan-out, but inside the route

Publish-Subscribe copies the message on the broker (SNS hands a copy to each subscribed queue). There's a close cousin that copies inside the route — Multicast: your step sends a copy of the exchange to several destinations itself.

From("sqs://orders")
    .Multicast()
        .To("sqs://orders-billing")
        .To("sqs://orders-shipping")
    .End();

Looks similar, but the difference is fundamental — where the copying happens, and who knows about whom:

	SNS→SQS (pub/sub)	Multicast
Where it copies	on the broker (SNS)	in the route (your process)
Who knows the recipients	SNS (the subscriptions)	the route (the `.To` list)
Adding a recipient	subscribe another queue, leave the publisher alone	edit the route's code
Coupling	fully decoupled	recipients are baked into the route
Parallelism / aggregating replies	none (fire-and-forget per subscription)	yes: `.Parallel().MaxParallelism(N)`, merge the branches

The rule of thumb: need to decouple the publisher from the subscribers (anyone can subscribe later, the publisher never knows) — SNS→SQS. Need to fan out to a fixed list right here and maybe collect the answers — Multicast (or its sibling Scatter-Gather from the Kafka post). One is broker topology, the other is a route step; you pick by where the fan-out logic should live.

Wrap-up

SQS and SNS in redb.Route are routes where the whole queue and the whole topic are one URI: From("sqs://…") / .To("sqs://…") / .To("sns://…"), with all the AWS SDK plumbing, the receive/delete dance, visibility, subscriptions and correlation tucked behind the string's parameters. Long-poll, FIFO, batching, competing consumers on a single parameter, at-least-once with a transactional ack, SNS→SQS fan-out with a choice of "envelope or bare payload" — the set is complete, and what you're left with is a short, legible string.

Straight talk on the trade-offs: there's no exactly-once (SQS itself doesn't offer it) — you get at-least-once plus idempotency; RPC isn't out of the box, it's assembled from attributes and .ToD; and an SNS subscription wraps your payload in an envelope by default. All of that is AWS being AWS, not the connector cutting corners — and the connector is upfront about every bit of it.

Two runnable examples — redb.Route/demos/SqsRpcDemo (RPC + concurrency) and redb.Route/demos/SqsPubSubDemo (SNS→SQS fan-out + raw delivery); both come up on LocalStack (http://localhost:4566, test/test) and run as a self-test. If something bites in your scenario, say so in the comments.

Sources and releases: github.com/redbase-app. About the redb database: redbase.app.

If this was useful — a ⭐ on GitHub helps others find it.

DEV Community: rinat kozin

redb 3.6.0: a bug report that turned out to live in six providers — plus AS2/EDI and a shared port

The defect was two layers below where it surfaced

The second defect in the same report: a tool with no context

AS2 and EDI, inside the route

One port for several connectors

Camel parity: four more things

IBM MQ: from half a second to single-digit milliseconds

Identity: e-mail uniqueness held by an index, not by a check

Tsak: a dead-letter store that did not work on PostgreSQL

Smaller things you would notice

Getting it

What is next

AS2 in .NET without a separate Java gateway: EDI with trading partners, right inside the route

AS2 in one minute

"Why not just HTTPS"

A readable route: the endpoint is a string

A partner is one object, not a scatter of parameters

One route, three environments

Sending (producer)

Receiving (consumer / AS2 server)

Asynchronous MDN

Algorithm matrix

How it differs from a gateway

Where it's used, and why a native connector

AS2 as one step of an end-to-end route

The full loop: order in, invoice out

Visible in one trace

What to know about AS2 in practice

Certificates and secrets

Interop proven, not claimed

FAQ

Install

A real local database in Blazor WebAssembly and MAUI: no EF DbContext, no Include, no migrations

Sound familiar?

What it looks like instead

The model for the examples

Which package to install

Quick start: mobile (MAUI)

Step 1. Project and package

Step 2. Registration in MauiProgram.cs

Step 3. Initialize exactly once

Step 4. The page

Quick start: Blazor WebAssembly

Wrinkle 1. The build needs an extra tool

Wrinkle 2. Initialization is manual here too

Wrinkle 3. Persistence is on you

Everyday operations

Create

Read by id

Update

Delete

Queries

Arrays

Add a field

The outbox queue

Making queries fast: filter on the base fields

Many-to-many without a join table

While we are here: trees

And there is quite a bit more

Syncing with a server that also runs RedBase

How it compares

Things worth knowing upfront

Wrapping up

A TMS backend, architecturally: cluster, coordinate streams and a typed object store

What the stack is made of

The unit of composition is a context, not a service

Physical topology

Why Redis is exactly six

Two zones

Transports and channels

The high-frequency stream: where the object model ends

One source, several aggregators

How the stream scales across nodes

The consequence everything else grows from

Planned route and actual route

The broker carries a key, not geometry

Reads fall back through three levels

Audit: one event, two roads

Partition maintenance

Step 2. Registration in `MauiProgram.cs`