redb 3.4.0: day-two operations for a .NET stack — replay what failed, hot-patch the framework, lock down the control plane

Building a system and running one are two different engineering problems. The first is done when it holds up under load. The second starts with the questions someone asks while holding a pager: what died overnight, and how do I re-run it? who hit force-stop? can I ship a one-line library fix without rebuilding the entire runtime? why is a service-account password sitting in plain sight on a dashboard page?

Previous releases of our stack answered the first question. 3.4.0 is entirely about the second.

Quick refresher on the cast: redb, a typed store on top of Postgres/MSSQL/SQLite; redb.Route, an integration engine in the Apache Camel spirit with 30+ connectors; redb.Tsak, a runtime with a dashboard, hot-reload and clustering; and redb.Identity, an OIDC / OAuth 2.1 server. All of it runs in our own production and ships as packages, images, and standalone archives.

Four things landed in 3.4.0, and every one of them is about the day after the deploy:

• Replay checkpoints — a save-point inside a route, so the tail can be re-run later from known-good state. This one goes all the way through: a primitive in the engine, a dead-letter queue in the runtime, a button in the dashboard.
• A shared runtime layer — the framework now lives beside the app instead of inside it. Patching a library is a file swap, not a release of everything.
• Secrets out of the logs — declarative redaction of endpoint URIs, plus named connection factories so the credential never enters the route to begin with.
• Roles and signed modules — role enforcement on the management API, a persistent audit trail, and a cryptographic trust anchor for code the runtime loads into its own process.

Quietly but importantly: redb.Identity moved onto the ecosystem version (1.2.2 straight to 3.4.0). One number across the stack, so "which Identity works with which Route" is a glance instead of a lookup table.

And as of 3.3.0, still true here: every Pro package is free across the whole 3.x line. No keys, no license server, no sign-up. dotnet add package redb.Postgres.Pro and it works.

Part of the redb / redb.Route series — recent posts first:

• The Telegram connector: a bot as a route
• redb 3.3.0: an enterprise .NET stack you actually own
• Leaving MassTransit for a Camel state of mind: the Kafka connector, Scatter-Gather, and transactions
• Apache Camel for .NET, dissected: the HTTP connector with no ASP.NET MVC + the Content-Based Router
• redb.Route — Apache Camel for .NET: 22 transports, 30+ EIP patterns, compiled DSL

Sources: github.com/redbase-app. About the database itself: redbase.app.

---

Part 1. Replay: a save-point inside a route

Starting with the feature I'm happiest about — because it runs through all three layers and looks natural in each one.

The problem everyone solves by hand

A five-step route. Step three charged the card. Step four blew up because the receipt service was down for ten minutes. Now what?

None of the usual answers are good. Retrying the whole route charges the card twice. Retrying from the exchange's current state is worse than it sounds: the body has already been reshaped by the steps in between and headers have been overwritten, so you're not replaying what happened — you're replaying whatever it decayed into on the way to the failure. Picking through it by hand at 3 a.m. isn't an answer at all.

The right model is a save-point: freeze the message exactly where things were still good, then re-run the tail of the route from that frozen state.

What it looks like in the DSL

From("timer://poll?period=5000")
    .Process(chargeCard)
    .Replayable("after-charge")      // save-point: the card is already charged
        .Process(sendReceipt)
        .To("http://receipts")
    .EndReplayable();

Everything after the marker is the tail. Each exchange passing through is snapshotted into exchange.Properties["route.checkpoint"] (last marker wins). There's a lambda form too, if you'd rather not track matching End calls:

.Replayable("after-charge", r => r
    .Process(sendReceipt)
    .To("http://receipts"))

Replay is a typed in-process call:

await context.ReplayAsync(routeId, "after-charge", snapshot, ct);

Declare the marker with exposed: true and it's additionally published as direct:__replay:{routeId}:{name}, so another route can simply .To(...) it. The default is private: a save-point shouldn't accidentally become a public entry point into the middle of your route.

Snapshot() — and why it isn't Clone()

This needed a new primitive in the engine, and it's the most interesting part.

IExchange has had Clone() for a long time. But Clone() deliberately shares the body with the original — Splitter aggregation depends on it, because the branches need to see the same object. For a save-point that's exactly backwards: if the body is shared, the next step mutates it in place and your "frozen" state was never frozen at all.

So IExchange.Snapshot() / IMessage.Snapshot() landed: a genuinely deep, isolated copy. v1 handles immutable bodies, byte[], and ICloneable, and throws loudly on anything else rather than quietly doing a shallow share. That part isn't negotiable — a silently shared body inside a save-point mechanism is the kind of bug that surfaces a month later, in production, on the one exchange you needed.

A snapshot carries no DI scope: it's dormant data headed for a database row, not a live exchange, so there's no per-message leak.

While we were in there, we also fixed the doc comment on Clone(), which had said "deep copy" for years while never being one. Both methods now describe what they actually do.

Where a checkpoint may and may not go

A save-point can't cross a branching composite (Choice, TryCatch) — the tail stops being well-defined, because which branch is the tail? Inside a long-running transaction it's allowed but warrants a warning: replay happens outside the original transaction.

We could have hard-coded a couple of ifs in the validator. Instead there's now a general mechanism, because constraints like this keep showing up:

• ICompositeScope / IDurableScope — scope category markers;
• IScopeNestingRule — a node declares Allowed / Warn / Forbid against an ancestor category;
• IBranchingDefinition — definitions whose children live outside Outputs (Choice When/Otherwise, TryCatch catch/finally) expose them for a generic tree walk.

The validator applies the rules uniformly: Forbid is a build error, Warn goes to the log. A new structural constraint now ships on the definition, never in the validator — which is precisely what Camel-style engines tend to get wrong, letting constraints sprawl across code that has no business knowing about them.

Layer two: the dead-letter queue in Tsak

A primitive in the engine is nice, but the person on call doesn't want an abstraction — they want the list of what failed overnight. So Tsak 3.4.0 puts a DLQ with replay on top of checkpoints.

CheckpointDlqHandler is a Tier-3 error handler installed on every module context. On failure it reads route.checkpoint off the exchange and dead-letters it. The key property: opt-in by construction. Only routes carrying a .Replayable() marker leave a checkpoint, so only those get captured. The queue never fights a broker or a transaction that already owns redelivery — it picks up exactly what you explicitly marked as "this needs a human re-run."

From there it's ordinary operational plumbing:

• tsak_dlq — a flat table (PG / MSSQL / SQLite), created at startup;
• ExchangeSnapshotCodec — serializes the snapshot for durable storage. byte[] and string round-trip exactly; everything else goes through System.Text.Json with the exact CLR type restored when its assembly is loadable. A non-serializable body is stored visible but flagged not-replayable — capture never breaks outright;
• API: GET /api/exchanges/failed (filtered, paged), POST /api/exchanges/{id}/replay, DELETE /api/exchanges/{id} — role-gated and audited;
• CLI: tsak dlq list | replay | discard;
• a Dead-letter dashboard page with server-side filtering and paging, so a large table never gets dragged into the browser;
• retention as a first-class cron://tsak-dlq-retention route on the system context (Tsak:Dlq:RetentionDays, default 30). Not a hidden timer buried in the runtime — a normal route, visible in the API, the dashboard, and the scheduler page. We eat our own cooking.

On semantics, plainly and without marketing: this is at-least-once and operator-driven. The tail may run more than once, so replayed side effects need to be idempotent. It is not Temporal-style durable execution and doesn't pretend to be. It's a tool for the person on call: show me what broke overnight, and re-run it once the fix is in.

---

Part 2. The shared runtime layer: patch the framework without a rebuild

The second big item is architectural, and it's about how fast you can react.

Before

Tsak takes your routes (.tpkg modules) and turns them into a service with a dashboard, metrics, and clustering. The framework itself — redb.Core(.Pro), the providers, redb.Route.* — used to sit in the app's bin folder. Which meant a patch to any library required rebuilding and re-releasing everything. One fix in the Kafka connector: rebuild Tsak, re-cut the archives, re-push the images, re-run the whole verification pass.

After

The framework moved to Libs/shared/ and is loaded from there at startup. Only redb.Tsak.* and redb.Licensing stay in the bin.

worker/
├── redb.Tsak.Worker.dll        ← the application
├── redb.Tsak.Core.dll
└── Libs/shared/
    ├── redb.Core.dll           ← the framework — swappable
    ├── redb.Postgres.Pro.dll
    ├── redb.Route.dll
    ├── redb.Route.Kafka.dll    ← connectors too
    └── runtimes/…              ← native dependencies alongside

The payoff: a binary-compatible patch to any leaf, provider, or connector — or a brand-new beta connector — ships by swapping a DLL in Libs/shared/. No rebuild, no re-spin of the Tsak and Identity archives. A 3.4.0 → 3.4.1 framework patch becomes a file copy.

How it works, briefly:

• Early bootstrap. SharedRuntime.InstallEarly is the very first statement in Program.cs, before a single redb type is touched. It installs the shared-layer resolver and byte-loads the framework — the file is never locked, which is exactly what makes it swappable on a live box. It reuses primitives we already had: tracker unification (so .tpkg modules still see one identity per redb type), a per-assembly native resolver (librdkafka, e_sqlite3), and version-tolerant forwarding.
• Fail fast. A missing or corrupt framework DLL in Libs/shared/ aborts startup immediately, naming the assembly and where it looked. Infinitely better than a MissingMethodException a day later under load.
• Compat gate. At startup the shared redb minor is checked against this Tsak build's minor. Patch drift is allowed — that's the entire point — but a minor mismatch, or a mix of minors inside Libs/shared/, stops the process.
• GET /api/system/assemblies (admin) — what redb is actually loaded: name, version, and origin (shared / bin / runtime). The diagnostic counterpart to a swappable layer, answering "which redb is running right now."

Tooling collapsed to one manifest (scripts/shared-manifest.psd1) and one parameterized scripts/build-shared.ps1 that serves both dev and publish. Plus a new scripts/refresh-shared.ps1: rebuild one library and drop its DLL into a target Libs/shared/ — your dev tree or an unpacked archive. The "patch without rebuilding Tsak" flow, as an actual command.

We verified it rather than assuming: 0 framework assemblies in the bin root vs 12 in Libs/shared; a clean start loads all 12 from the shared layer and boots (SQLite Pro, cluster, scheduler); Kafka with librdkafka, Mail with MailKit, and SQLite with e_sqlite3 all served through the shared layer and exercised end to end; .tpkg modules unify; the negative case aborts with a clear message.

Planned maintenance: cordon / uncordon

Same "day after the deploy" theme, and it's in Pro — so, free.

A cluster node used to have two states: working, or gone. That's not enough for a rolling upgrade: rebalance reshuffles everything and remove-node is a hard eviction, when what you want is the middle state — this node finishes what it's doing and takes on nothing new.

tsak cluster cordon   node-3     # takes no new work, drains its route locks to peers
# … upgrade the node …
tsak cluster uncordon node-3

The Cordoned flag is durable and orthogonal to status: the node stays Online, because it is alive — it's just closed for new business. The per-route watch loop reads a process-local mirror of the flag, refreshed from the node's own record on each heartbeat, stops acquiring route locks, and releases the ones it holds so peers can pick them up. Work in flight drains normally. There's an API (POST /api/cluster/nodes/{id}/cordon and /uncordon, audited), a client method, the CLI above, and Cordon/Uncordon buttons on the dashboard's Cluster page.

If you've spent time around Kubernetes, both the semantics and the word will look familiar. That's deliberate: don't invent new vocabulary where the industry already has one everybody knows.

---

Part 3. Secrets: don't mask them, don't accept them

The third block is operational security — and the interesting part isn't what got fixed, it's how the approach changed.

Declaration beats guesswork

In an engine where endpoints are URIs (ldap://…?bindPassword=…), sooner or later you have to decide which parts of that URI may reach a log. The classic answer is a deny-list of parameter names: password, secret, apiKey…

A deny-list fails open. Add a new option carrying a credential and it goes to the logs in the clear until somebody remembers to extend the list. The failure is silent: you find out by reading logs, not by failing a build.

In 3.4.0 the source of truth is a declaration on the option itself:

public class LdapEndpointOptions : EndpointOptions
{
    public string Server { get; set; } = "localhost";   // printed in logs

    [Sensitive]
    public string? BindPassword { get; set; }           // always ****
}

EndpointOptions.BindFromUri harvests those declarations by reflection (once per options type) and feeds them to the URI sanitizer. The set of secret keys is derived from the code and never hand-maintained — a new credential option can't be forgotten, because the only thing you could forget is the attribute, and the attribute sits right there on the property.

This is exactly how Apache Camel does it: @UriParam(secret = true) is the declaration, and the runtime list is generated from those annotations by a build plugin. The .NET version needs no build step — reflection is enough. All 37 credential options across 22 connectors are annotated. The name-based heuristic survives as a backstop for a URI rendered before any endpoint of that scheme exists.

Formatting is preserved: the new EndpointUri.Sanitize(string) keeps the scheme, ://, path, parameter order, and every non-secret value byte for byte, replacing secrets with a constant ****. Every boundary now goes through it: route-build and endpoint-start logs, the redb.route.endpoint OpenTelemetry span tag and metric label, in-flight exchange and health-check metadata, and the CompiledRoute.FromUri DTO the CLI and dashboard render. Unnamed routes now derive a sanitized route id too, so a secret can't leak out through a {RouteId} log line.

Also in this pass: userinfo passwords are finally inside the fence (amqp://user:pass@host → user:****@host); redb.Route.Elasticsearch sanitizes node URLs in its startup log; and redb.Route.Exec logs the executable plus an argument count — command-line argument values are secrets far too often to log on principle.

There's a public API for connector authors as well: EndpointUri.Sanitize(string), IsSensitiveKey(string), and AddSensitiveKeys(params string[]) — the direct analogue of Camel's addSanitizeKeywords.

Separately, EndpointUri.RedactSecrets(string) handles arbitrary text: a driver exception is perfectly capable of dragging a connection string with Password=… into ex.Message, and OnExceptionProcessor logs that message on redelivery and on retries-exhausted. Both sites now run it through the redactor. One honest limitation worth knowing: with LogStackTrace enabled, the exception object itself is handed to the logger and can't be scrubbed in-process — that path needs a filter in your logging sink.

Better than masking: never take the secret

Masking is defense. Offense is making sure the credential never enters the URI at all. So 3.4.0 rolls out a named ConnectionFactory across the ten connectors that had no such mechanism: Telegram, MQTT, HTTP, Mail, FTP, SFTP, SignalR, gRPC, TCP, WebSocket.

context.AddToRegistry("support-bot", new TelegramConnectionFactory {
    Token = Environment.GetEnvironmentVariable("TELEGRAM_TOKEN")! });

r.From("telegram://receive?connectionFactory=support-bot")   // no token in the route

No secret in the URI means nothing to mask in logs, telemetry, or the dashboard. The factory fills in only the options the URI didn't set explicitly, so an inline URI value always wins and existing routes are untouched; a name that isn't in the registry logs a warning and falls back to URI parameters.

One detail I'm prouder of than the feature itself. For connectors whose address lives in the endpoint path (HTTP, Mail, SignalR, gRPC, TCP, WebSocket) the factory deliberately carries no host or port. Otherwise an innocent typo in a factory name could silently redirect a route somewhere else. For the same reason the wss scheme still forces TLS on regardless of what the factory says. A mechanism for credentials must not quietly become a mechanism for routing.

redb.Route.Ldap got an LdapConnectionFactory in the same pass — and with it, a route that carries no credentials whatsoever:

context.AddToRegistry("honest-ldap", new LdapConnectionFactory {
    Server = "ldap.corp.local", Port = 636, Ssl = true,
    BindDn = "cn=svc-reader,dc=corp,dc=local",
    BindPassword = Environment.GetEnvironmentVariable("LDAP_BIND_PASSWORD") });

r.From("ldap://SEARCH:dc=corp,dc=local?connectionFactory=honest-ldap&filter=(objectClass=user)")

One caveat to internalize: CompiledRoute.FromUri is now a display value. Routing identity, endpoint cache keys, and message flow didn't change by a single byte, but parsing FromUri to recover credentials is no longer a thing you can do. It's a display, not a data source.

---

Part 4. Who can do what: roles, signed modules, audit

The management API is, functionally, the keys to prod. In 3.4.0 the keys got locks.

The role model

Tsak API keys have carried roles since 1.0.0. In 3.4.0 those roles started being required at the endpoints — until now the viewer < operator < admin ladder existed in the model, but no endpoint declared a requirement, so a key issued for dashboard access could do everything an admin key could.

It's declarative, in the same style as everything else here:

• RequiresRoleAttribute — on an action or a whole controller; multiple roles are OR-ed; a method-level attribute overrides the controller-level one;
• NoRoleRequiredAttribute — for technical endpoints that must never start answering 403;
• TsakRoles — the ladder itself, with reader/ops synonyms; custom roles match by exact name only;
• RoleAuthorizationProcessor — the enforcement, wired into the system pipeline immediately after a successful auth check. It resolves the target action through the same ControllerRegistry the dispatcher uses, so both agree on where the request would have landed.

How it shook out: admin covers all of /api/auth/* (reads included), /api/users/*, deleting a context or a module, route force-stop, cluster rebalance and node removal. operator covers diagnostics and logs (both expose internals) plus every mutating endpoint by default. viewer covers every other GET.

A specific concern here was not breaking probes. The check runs only for authenticated exchanges, so auth-exempt Kubernetes probes pass straight through; HealthProbeController is additionally marked [NoRoleRequired], so even if an operator narrows Tsak:Api:AuthExempt, the probe can't start returning 403 and take the deployment down with it. The echo and Prometheus routes have their own pipelines and never reach the check at all.

Compatibility is staged, as it has to be for something like this. Nothing changes when auth is disabled. Keys with no roles keep full access and log a one-time warning — then, once every key has been reissued with explicit roles, you set Tsak:Auth:RolelessKeysAreAdmin=false and close the door. Enforcement as a whole can be switched off with Tsak:Auth:EnforceRoles=false.

Coverage: 32 tests on the processor and the ladder, plus 27 integration tests driving the real Kestrel pipeline with viewer, operator, and roleless keys — including proof that probes answer without a key and never return 403.

A module is code. So: signatures

Deploying a module used to mean filesystem access. Now modules can be uploaded and rolled back over the API — but since a module is code Tsak loads into its own process, the whole feature is built around a trust anchor:

tsak module keygen              # generate an ECDSA key pair
tsak module sign  my.tpkg       # → my.tpkg.sig
tsak module validate my.tpkg    # dry run: same checks as upload, installs nothing
tsak module deploy my.tpkg      # upload (signature in a header)
tsak module rollback my-module  # restore the previous version

The parts that matter:

• Upload is off by default (Tsak:Modules:Upload:Enabled=false) — a node that doesn't need remote deploy exposes no upload surface at all.
• ModuleSignatureVerifier does RSA/ECDSA detached-signature verification with the BCL only. No cosign dependency.
• Enforcement at the load boundary: with Tsak:Modules:Signature:Required=true and a configured public key, every .tpkg — uploaded or dropped into the directory by an operator — must carry a valid .tpkg.sig or it's refused before any of its code loads. The trust anchor becomes the public key, not filesystem access. That's stricter than the WSO2 MI default.
• Upload-time guards: a size ceiling, valid-ZIP and manifest checks, the stored name taken from the manifest and sanitized (no /, \, .. — path-traversal and zip-slip safe), fail-fast signature verification, atomic install (temp → move), previous version archived.

Related: staged validation before hot-swap. A package update used to tear down the running version before opening the new one — so a broken package left the context with no modules at all. Now the new .tpkg is first opened in a throwaway collectible ALC (without mutating the shared assembly tracker, so running modules are untouched) and checked that it loads and discovers at least one module. Only then is the old version torn down. A package that won't open, or that has no modules, is refused with a logged reason — and the current version keeps running.

An audit trail that survives a restart

Admin actions used to go to the log, with lifecycle history in a 1000-entry in-memory ring — so after a restart there was no record of who did what. Now there's tsak_audit_log: a flat table (deliberately not a redb object — it's append-only, it grows, and its schema is fixed), created at startup for the configured provider, exactly the way the Quartz tables already were.

It's written through a direct://tsak-audit route (Sql.Execute INSERT) — and that's a feature, not showing off: because the sink is an endpoint, the same event stream can be pointed at a file, a broker, or an HTTP collector by configuration alone, with no new code. Writes are fire-and-forget through a bounded queue with a background pump: an API call never waits on the database, a broken audit backend can never take the node down, a backend failure falls back to the log sink, and sustained flooding drops the oldest queued events with a warning.

Reading is GET /api/audit (role admin, filtering and paging entirely server-side), the tsak audit CLI command, and a new Audit dashboard page. Retention is — again — a normal cron://tsak-audit-retention route (Tsak:Audit:RetentionDays, default 90; 0 keeps forever).

For deployments with no database, audit stays in the log but now emits a [tsak-audit]-anchored line with one JSON object right after it, so a standalone node can be grepped and parsed without any log-format wrangling.

Get paged, don't poll

Last of the operational block: the watchdog detected hung and suspected exchanges, but the alerts just piled up behind GET /api/watchdog/alerts — a poll nobody performs at 3 a.m. Now they're pushed.

AlertDispatcher fans each new alert out to every enabled channel, fire-and-forget (bounded queue plus a background pump, same as the audit sink): the watchdog scan never blocks on a slow SMTP server. Dedup is by context + route + exchange + level within DedupWindowMinutes — the scan rebuilds its snapshot every cycle, so without it a single hung exchange would page you on every tick.

Channels, all off by default: webhook (Slack / Teams / PagerDuty / any collector), telegram (Bot API over plain HTTPS, no connector), email (SMTP via the BCL), and endpoint — the generic one: send to any redb.Route producer URI (kafka:, rabbitmq:, amqp:, sqs:, mqtt:…) through a ProducerTemplate. That last channel covers every broker with zero per-connector code, and because the host supplies the component, no broker ever becomes a compile-time dependency of Core.

You can verify the wiring without waiting for a real outage: POST /api/watchdog/test-alert sends a synthetic alert through every enabled channel and returns the per-channel outcome, bypassing the dedup window. In the dashboard it's a "Send test alert" button on the Watchdog page's Alert Delivery panel.

---

Part 5. The core: a scheme name becomes your decision

Down to the storage layer. The headline change is small on paper and very visible in practice.

[RedbScheme(Name = "...")]

A scheme used to be named after the CLR type's FullName, with the string in the attribute serving as a cosmetic alias. The consequence is familiar: a scheme's identity in the database was welded to your namespace and class name. Rename the class or move the namespace, and you either don't refactor or you go fix the database by hand.

[RedbScheme("Customer note", Name = "Notes.Note")]  // positional arg is still the alias
public class Note { … }

The positional argument is, and stays, the alias — an explicit name is only settable through the named Name parameter, so not one existing declaration changes meaning, and types without Name behave exactly as before.

On the first sync, a type with an explicit name has its scheme renamed in the database. The lookup walks a three-step chain — explicit name, FullName, short type name — and the first match is renamed in place. Physically that's a single-row UPDATE of _schemes._name: the id is preserved, objects, structures and values are untouched, and polymorphic loading (which resolves through scheme_id) never notices.

The honest warning, which is in the changelog and belongs here too: renaming requires updating every consumer of that database together. An app version that predates the explicit name won't find the scheme under its new name and will create a second one, silently splitting objects across both. If that has already happened, redb now detects it and refuses to continue rather than picking one at random.

Names are validated against C# identifier rules (Latin letters, digits, _, ., +; no reserved words; 128 chars max) in C#, before any SQL is issued, so the error names the offending type. And they're all validated up front and reported together in a single AggregateException — fixing a codebase one name per restart would have been cruel. Human-readable titles still live in Alias, which is free-form.

Provider parity got closed in the same pass: MSSql and SQLite now carry a _schemes name-validation trigger mirroring the PostgreSQL rule for rule, so a name accepted by one provider is accepted by all three. And a scheme's _alias now syncs on every sync (structures always did): the attribute is the source of truth, removing it resets _alias to NULL.

Load checks the scheme

LoadAsync<TProps> now verifies the object's _id_scheme against the scheme TProps maps to — before anything reaches the cache.

By default a mismatch returns null. That's not an arbitrary pick: a soft-deleted object gets scheme -10, and soft-delete callers expect exactly null back. If you'd rather have a genuine type mistake be loud, set RedbServiceConfiguration.ThrowOnSchemeMismatch = true and get a RedbSchemeMismatchException instead. The untyped LoadAsync(objectId) is unaffected either way.

Several nodes starting at once

A scenario reproduced on a three-node cluster: several instances start against a database that doesn't have a given scheme yet, all of them miss the lookup, and all of them issue INSERT INTO _schemes. Exactly one wins.

Scheme creation now uses a conflict-free statement per dialect (ON CONFLICT (_name) DO NOTHING on PostgreSQL and SQLite, INSERT … WHERE NOT EXISTS with UPDLOCK, HOLDLOCK on MSSql), and whoever loses the race reads back the winner's row.

The detail that makes this more interesting than it looks: catching the unique violation wouldn't have worked. On PostgreSQL a failed statement inside a transaction poisons it, so the follow-up read fails with 25P02. The right answer isn't "handle the exception" — it's "don't create the conflict." Both creation paths are covered, typed and untyped.

Pro data migrations now run on every provider

The Pro data-migration mechanism works across all three providers and is covered by tests (MigrationTestsBase: apply, history row, idempotency, dry run, against all three Pro fixtures). Getting there sorted out four separate things: how the executor obtains the DB context; dialect-aware UPDATE generation (ISqlDialectPro.Migration_UpdateTarget — SQLite forbids aliasing an UPDATE target, T-SQL wants the alias bound in a trailing FROM); a SQLite-native _migrations DDL (with a REAL Julian _applied_at, per SQLite's own time convention in redb); and dropping IDENTITY from _migrations._id on MSSql, since the executor supplies ids explicitly like everywhere else in redb.

Existing databases don't pick up the new _migrations DDL — initialization is skipped when _schemes already exists. In practice that breaks nothing, since there's no migration history there to preserve.

Small things you'll feel

• Hot-reload no longer pins memory. The process-wide schemeName → Type index held strong Type references, and a Type keeps its AssemblyLoadContext alive — so a collectible ALC (the thing Tsak's hot-swap is built on) could never be collected, and after a reload the stale instance produced a false name conflict against the fresh one. Entries are now WeakReference<Type> with dead ones pruned on lookup, and two instances of the same FullName from different ALCs are recognized as a reload rather than a clash. Distinct types sharing one explicit Name still conflict, by design.
• redb cache options are reachable from Tsak config — the Tsak:Redb:Cache section (props cache and TTL, list cache, metadata cache, AutoRecomputeHash, cache domain). A Tsak node used to run on redb's defaults because only two settings were passed through. Every key is optional and an absent section changes nothing, and the whole set with default values is spelled out in appsettings.json so the knobs are visible and editable in place. Enabling SkipHashValidationOnCacheCheck together with clustering logs a startup warning: trusting the cache without re-checking the object hash is single-writer territory.
• 679 lines of dead cache-interface layer removed — five interfaces that referenced only each other, with no implementations and no consumers. The live caches are GlobalMetadataCache, GlobalListCache, GlobalPropsCache, and the type index. Technically a breaking change since the types were public; practically unusable, since nothing implemented them. The caching README.md was rewritten to describe the caches that actually exist.

---

Part 6. redb.Identity: one version, JAR, and an honest conformance answer

Why 1.2.2 → 3.4.0

Until this release redb.Identity rode its own 1.x line while core, Route, and Tsak moved together on 3.x. Two numbering schemes turned "which Identity works with which Route" into a lookup instead of a glance.

From 3.4.0 on, Identity shares the ecosystem version and jumps straight to the shared number — the same redb.Route 3.4.0 it's built against. Every future ecosystem release bumps it too.

To be explicit: this is a version realignment, not a semver-breaking release. There are no breaking API changes here, and the one new feature is off by default. The major digit changed because the number is now the ecosystem's, not because Identity's contract broke. The old 1.0.1–1.2.2 tags remain valid history.

JAR (RFC 9101) — signed authorization requests

/connect/authorize now accepts a signed request object, gated behind Features.EnableJar and off by default, so every prior release behaves identically (a request parameter still gets request_not_supported).

When it's on: a signed request (inline) or request_uri (by reference, fetched over HTTP) is verified against the client's registered keys, and the parameters inside the JWT take precedence over the query string, per §6.1.

What is not accepted: alg: none (an unsigned request object destroys the exact integrity guarantee JAR exists for), a wrong-key signature, a mismatched inner client_id, an expired object, and — under Enforce — an algorithm that disagrees with the client's declared RequestObjectSigningAlg. All of them come back as invalid_request_object.

Rollout is staged: Off → LogOnly → Enforce. Discovery advertises request_parameter_supported, request_uri_parameter_supported, and the algorithm list only when JAR is on — a server shouldn't promise in metadata what the endpoint won't actually do. Configuration lives in the Jar section: enforcement mode, allowed algorithms (asymmetric only), clock skew, size limits, and the SSRF knobs for request_uri.

Now the part that was most of the work. OpenIddict 6.3.0 does not support request objects. We checked before planning: the assembly carries ValidateRequestParameter and the request_not_supported string, but nothing that parses a request object or verifies its signature, and request_uri exists only as a PAR URN. So JAR here isn't a config flag — it's our own server handlers (ValidateRequestObjectHandler) that slot in ahead of the built-in unconditional rejection and do the work themselves. PAR-issued urn:ietf:params:oauth:request_uri:* identifiers are deliberately left alone: PAR resolves those itself, and a JAR handler has no business there. request_uri fetches are size- and timeout-bounded and pass the SSRF filter below.

Coverage: 15 handler tests plus a live conformance demo, demo_jar_request_object.ps1, wired into the run_all sweep — so the scenario is exercised against a running server, not just a unit test.

What's not here yet: request-object encryption (JWE) is deferred — the client's RequestObjectEncryptionAlg / ...Enc are stored but not applied; that's a separate phase. RequestObjectSigningAlg, previously stored and ignored, is now genuinely enforced under Enforce, and JwksUri is genuinely resolved. The full six-phase plan, with risks (SSRF, alg:none, algorithm substitution) and sizing, is in the repo at doc/JAR_RFC9101_PLAN.md.

Resolving client keys

Underneath all this sits IClientKeyResolver — the resolver for the keys that verify what a client signed: a JAR request object today, a private_key_jwt assertion later. Two sources: the inline JsonWebKeySet on the application, or JwksUri, fetched and cached with background refresh on a TTL plus a rate-limited forced refresh on a kid miss, so a client rotating its keys doesn't break sign-in until the TTL expires.

Two principles, both about not letting uncertainty turn into permission:

• Fails closed. An unreachable or malformed JWKS yields no keys, so there's nothing to verify against and the caller rejects. "Couldn't check" is never treated as "valid."
• A broken inline JWKS never falls back to jwks_uri. A typo in a pasted key set has to stay visible instead of being papered over by a silent switch to another key source.

Asymmetric algorithms only: ClientSecret is stored as a BCrypt hash and verifying an HMAC needs the original secret, so HS* is impossible here on mechanics alone — and FAPI 2.0 forbids it for request objects regardless. Resolver settings live in the ClientKeys section: cache lifetime, minimum refresh interval, fetch timeout, maximum document size, plus RequireHttps and AllowPrivateNetworkTargets — two development-only relaxations that have no business in production.

The SSRF guard

Anything arriving from outside as a URL (jwks_uri, request_uri) goes through OutboundUrlGuard — before a socket is opened. It rejects anything that isn't an absolute HTTPS URL, and anything resolving to a non-public address: loopback, RFC 1918, link-local — including 169.254.169.254, the cloud metadata endpoint that hands instance credentials to whatever can reach it — CGNAT, IPv6 unique-local, and the IPv4-mapped forms like ::ffff:10.0.0.1 that a naive check walks straight past.

40 tests, 29 of them on the guard alone — including one asserting that 172.32.* and 172.15.* are public and must not be blocked (the 172.16/12 boundary is a classic off-by-one). All of them stay offline: attempting a network call fails the test by itself. Stating the known limitation myself: this does not close DNS rebinding, which needs the validated address pinned onto the connection itself.

About conformance — why SKIPPED is the right answer

After enabling JAR, the local OpenID Foundation Basic OP run went from 1 SKIPPED / 4 REVIEW to 2 SKIPPED / 3 REVIEW, with 0 FAILED throughout.

That deserves an explanation, because the number reads worse than the reality. Both SKIPPED modules test the unsigned (alg:none) request object. The suite skips them when the server doesn't advertise none among its supported algorithms — which is precisely our position: we support signed request objects only. Better still, one module moved REVIEW → SKIPPED because the server now genuinely processes signed request objects and advertises that honestly, instead of the previous no-support state that sent the test down a screenshot path.

There's exactly one way to turn either skip into a pass: start accepting alg:none. We're not trading a security property for a nicer line in a report. The full write-up is in OPENID_CERTIFICATION.md §4.3, so anyone looking at the badge sees the same picture we do.

---

Part 7. Connector odds and ends

Briefly, so it doesn't get lost. The Telegram connector (which got its own post) picked up some ergonomics:

• replyToMessageId as a first-class producer option, expression-capable, with ReplyTo(long) / ReplyTo(IExpression) / .ReplyToIncoming(). Replying to the message that triggered the exchange no longer needs a hand-written .Process step copying one header into another.
• messageId for edit/delete — same story: send-then-edit is now Tg.Edit(token).MessageId(Header(TelegramHeaders.SentMessageId)).
• showAlert in answer mode — the callback answer shows as a modal alert instead of a toast.
• A telegram.caption header for document/photo that wins over the option, consistent with how parseMode and fileName already behaved.
• Mini App payloads (WebApp.sendData) now reach the route: the data becomes the exchange body — same contract as text messages and callback queries — and is also exposed as headers. Works on both the long-polling and webhook paths (they share a mapper); filter on telegram.messageType = "WebAppData".
• And a fix: document / photo modes now genuinely pass telegram.replyToMessageId and telegram.replyMarkup through, so a photo with inline buttons keeps its keyboard.

---

Upgrading from 3.3.x

The release is backward compatible and existing routes are unchanged. Three things are worth a look, though.

1. The SQLite native extension was renamed: redb → redbsqlite. The package now ships runtimes/<rid>/native/redbsqlite.{dll,so} (win-x64, linux-x64, linux-arm64). The reason is mundane: the generic name collided with the managed redb.* assemblies and, worse, matched the redb.* prune globs a host applies to its own bin — a native loadable module was getting swept up as if it were one of ours. The C init symbol is unchanged (sqlite3_redb_init) and the binaries are byte-for-byte the ones shipped in 3.3.3 — this is a file rename, not a rebuild.
   • Nothing to do if the package resolves the path for you (SqliteDataSource.LocatePackagedExtension(), the default for the Free DI registration) — it looks for the new name.
   • Action required if you pin the path yourself: REDB_SQLITE_EXTENSION, an explicit NativeExtensionPath, a Dockerfile COPY, or a deploy script that copies redb.so by name. A stale path fails at connection open, not at build — i.e. at runtime.
2. Tsak: roles are now enforced. Roleless keys keep full access by default (with a one-time warning), so nothing should break on day one. Reissue keys with explicit roles, then set Tsak:Auth:RolelessKeysAreAdmin=false. Check that your CI scripts use a key with the right level: route force-stop and module deletion are now admin, diagnostics and logs are operator.
3. Tsak: the distribution build changed. The framework is now built into the shared layer: scripts/build-shared.ps1 -IncludeFramework (publish does this for you via publish/build.ps1). Running the Worker without it hits the fail-fast, and the message tells you exactly what to run. Prebuilt 3.4.0 images and archives are already correct.
4. Implementing IModuleHealthContributor? It moved from redb.Tsak.Core to redb.Tsak.Contracts: change using redb.Tsak.Core.Contracts; to using redb.Tsak.Contracts;. Nice side effect — your module can now reference only the lightweight contracts assembly and drop the redb.Tsak.Core reference entirely.

---

Getting it

Core and providers from NuGet:

dotnet add package redb.Core
dotnet add package redb.Postgres      # or redb.MSSql / redb.SQLite
dotnet add package redb.Postgres.Pro  # Pro — free, no key

The engine and whichever connectors you need:

dotnet add package redb.Route
dotnet add package redb.Route.Telegram
dotnet add package redb.Route.Kafka

The runtime as an image or an archive:

docker pull ghcr.io/redbase-app/redb-tsak-stack:3.4.0
# or the standalone archive from the v3.4.0 GitHub release (linux-x64 / win-x64), cosign-signed

Images are cosign-signed; the public key ships with the release (cosign.pub):

cosign verify --key cosign.pub ghcr.io/redbase-app/redb-tsak-worker:3.4.0

Everything targets .NET 9.

---

Wrapping up

If 3.3.0 was about the stack holding up under load, 3.4.0 is about the stack being operable.

What failed gets replayed from a save-point instead of from whatever the state decayed into on the way down — and it's a button in a dashboard, not a night spent reading logs. A library patch ships as a DLL swap, with no runtime rebuild and no re-cut archives. A node leaves the rotation gracefully instead of being evicted. Secrets stay out of logs, telemetry, and the dashboard — and better yet, never enter the route. The control plane hands out permissions by role, module code is verified against a signature before it loads, and the record of who did what survives a restart. A scheme's identity in the database stopped being hostage to your namespace. And the whole ecosystem, Identity included, finally rides one version number.

This is the class of capability nobody sees in a demo and everybody feels in production.

If you take it for a spin, tell me what landed and what's missing — deep dives on replay checkpoints and on the secret-redaction work are both queued up, and I'd rather aim them at real questions.

Sources and releases: github.com/redbase-app. About the database: redbase.app. Previous posts: my dev.to profile.

If this was useful — a ⭐ on GitHub helps others find it.