DEV Community: Kingsley Onoh

Confidence Is Not Ownership

Kingsley Onoh — Wed, 06 May 2026 16:00:28 +0000

What should a finance queue do when two credible records point at the same case?

An invoice discrepancy and a contract breach can describe the same dispute. They can also describe two different disputes with the same counterparty, the same currency, and nearly the same amount. That is the trap in a finance operations queue. The data looks related before anyone has proved ownership.

The Workbench ingests exceptions from invoice reconciliation, transaction reconciliation, contract lifecycle events, webhook dead letters, manual operator entry, and signed Hub fanout. Every source carries its own identifiers. Some are reliable. Some are only reliable inside the upstream tool that produced them.

I had to decide what the system should do when a new exception looks like it belongs to an existing dispute.

The tempting version is simple: compute a score, pick the highest dispute, attach the exception. That makes demos feel clean. Exceptions flow in, disputes become richer, and the queue stays small.

It is also how a finance system quietly corrupts its own audit trail.

Once an exception is attached to a dispute, every later action inherits that fact. SLA timers, resolution playbooks, Notification Hub events, audit PDF exports, and operator comments all treat the relationship as true. If the relationship was only probable, the system has converted probability into evidence.

That conversion is the real design problem.

The Scoring Shape

The correlator in src/drw/domain/correlator.clj uses seven signals. Source reference and entity id each carry 0.15. Counterparty carries 0.25. Currency carries 0.10. Amount carries 0.15. Date carries 0.10. Category carries 0.10.

Those weights are not magic in the sense of being secret. They are visible because this is a spec project. But the structure matters more than the numbers.

Counterparty is the gate. A candidate dispute is not eligible unless it belongs to the same tenant, is not terminal, has the same counterparty, and falls within the correlation window. Only then does scoring begin.

That means the correlator is not a general similarity search. It is a tenant-scoped dispute ownership test.

The amount signal has a 10 percent tolerance and only scores when the currency also matches. The date signal checks whether the exception was observed within 72 hours of the dispute creation time. The source reference and entity id signals compare against exceptions already attached to the candidate dispute, not just fields on the dispute itself.

That last part matters. A dispute becomes easier to recognize as it accumulates evidence. The first invoice mismatch may create the dispute. A later webhook dead letter with the same upstream reference can now match the attached evidence, even if the dispute record itself does not carry that reference.

The core function is plain Clojure:

(defn score-candidates
  ([tenant-id exception disputes attached]
   (score-candidates tenant-id exception disputes attached {}))
  ([tenant-id exception disputes attached opts]
   (let [cfg (merge default-config opts)
         review (get-in cfg [:thresholds :review])]
     (->> disputes
          (map-indexed vector)
          (filter (fn [[_ dispute]]
                    (candidate-eligible? tenant-id exception dispute cfg)))
          (map (fn [[idx dispute]]
                 (assoc (score-candidate tenant-id exception dispute attached cfg)
                        :sort-index idx)))
          (filter #(>= (:score %) review))
          (sort-by (juxt (comp - :score) :sort-index))
          (mapv #(dissoc % :sort-index))))))

There are two details in that function I care about.

First, eligibility happens before scoring. A cross-tenant dispute receives no score. A terminal dispute receives no score. A different counterparty receives no score. The function does not let a high amount or date match compensate for a broken boundary.

Second, ties preserve input order through :sort-index. That is not glamorous. It prevents unstable review queues where two equal candidates swap positions between renders and make operators think the system changed its mind.

What I Was Wrong About

I initially treated the correlation score as the hard part.

It was not. The harder question was what the system does with the score.

There are three possible outcomes in src/drw/domain/exceptions.clj. If no candidate passes review, the exception creates a new dispute and attaches immediately. If the best candidate hits the auto-merge band and auto-merge is explicitly enabled, the exception attaches and records an auto-merged correlation. Otherwise, the system creates pending correlation records and emits a dispute.correlation_pending event.

That middle branch is intentionally hard to reach. The .env.example values set AUTO_MERGE_THRESHOLD=0.92 and REVIEW_THRESHOLD=0.70, while the source correlator defaults are lower for unit-level behavior. The runtime config is stricter because this is finance operations. False attachment costs more than a larger review queue.

What surprised me is that pending correlation became a domain object, not a UI convenience.

The queue needed an id, a tenant id, an exception id, a target dispute id, a score, a rationale, a status, a decided-by user, and decision timestamps. That is a lot of structure for something that could have been a modal row.

But the moment an operator accepts or rejects a candidate, that decision becomes part of the case history. A rejected match is useful evidence. It says someone looked at the overlap and decided the exception did not belong there. If the same upstream source sends a related item later, the prior rejection explains why the system did not combine the cases earlier.

That is why correlation records live next to exceptions and disputes instead of inside a transient UI response.

The Failure Mode Hidden In Good Matches

The most dangerous false match is not a ridiculous one.

It is the match that looks reasonable.

Same counterparty. Same currency. Amount within 10 percent. Observed inside three days. Category is billing. If those signals point to the wrong open dispute, the system does not look broken. It looks efficient.

The damage appears later. A Workflow Engine playbook starts against the wrong case. A Notification Hub event tells an operator that the dispute is ready for resolution. The audit PDF now contains an exception that belongs somewhere else. Nobody sees the root mistake because every downstream artifact is internally consistent.

That is the kind of bug that worries me more than a 500 response.

A 500 stops the flow. A wrong attachment keeps moving.

The design answer was to make confidence create a decision, not mutate the dispute. A review-band candidate becomes work for an operator. The UI exposes accept and reject actions. The API carries the same boundary. The audit log records correlation creation and later decisions.

The Workbench still supports auto-merge, but it is a policy choice. It has to be enabled. The score has to clear the higher band. The code does not pretend the existence of a scoring function means the business has accepted the risk.

Why Tenant Scope Belongs Inside The Algorithm

Tenant isolation is usually discussed at the HTTP layer. API key comes in, tenant id gets attached to the request, handlers filter queries.

That is necessary, but it is not enough here.

Correlation is a cross-entity operation by nature. It compares a new exception against many existing disputes and attached exceptions. If the algorithm accepts a list that accidentally contains another tenant's disputes, the HTTP layer is already too far away to save it.

So score-candidate checks tenant equality itself. score-candidates filters candidates through candidate-eligible?, which repeats tenant, status, counterparty, and time-window checks before scoring.

This is defensive duplication with a purpose. The route should pass tenant-scoped collections. The domain should still reject anything outside the tenant boundary. In a single-tenant fixture, this looks redundant. In a two-tenant test, it is the difference between "the route behaved" and "the invariant held."

The same philosophy appears in reports. The audit PDF renderer captures a tenant snapshot, renders with strict token lookup, and the setup check renders two tenants to make sure one tenant's identity literals never appear in the other tenant's output.

The theme is the same: do not trust a boundary because a previous layer probably handled it.

The Result

The finished local build passed 160 tests with 869 assertions. The full-flow E2E drives invoice adapter polling into exception creation, assignment, investigation, Workflow Engine resolution polling, Notification Hub event capture, and audit PDF generation.

The number I care about most is smaller: the dashboard guard. It caught a practical operations failure. The first dashboard shape rendered every dispute link in the tenant fixture. The fix capped the overview at 50 open disputes and kept totals intact.

That is the Workbench in miniature. Preserve the facts. Limit the surface. Make the operator decide when the machine only has a probability.

Confidence is useful. Ownership is a human or policy decision.

Why I Made WebSocket Delivery the Disposable Part of the Tracking System

Kingsley Onoh — Wed, 06 May 2026 14:53:35 +0000

The uncomfortable failure is not a carrier outage. That one is loud. The polling loop logs it, retries it, and moves on.

The failure I had to design around is quieter: the gateway receives a real carrier update, writes it to the database, then Redis is down at the exact moment the WebSocket fanout should happen. The client misses the live push. The shipment state is still correct. The event timeline is still correct. But the thing the user was watching in real time never moves.

That sounds like a broken real-time system until you decide which part is allowed to be temporary.

I made the database the truth and the WebSocket stream the delivery layer. That one decision shaped the rest of the gateway.

The real boundary

DHL, DPD, and GLS do not send one clean stream of facts. The adapters have to handle different request shapes, different status codes, and different timestamp rules. DHL uses DHL-API-Key and returns shipment events under shipments[0].events. DPD can return HTML for an error response, so DpdAdapter checks the content type before it ever tries to parse JSON. GLS sends a date like 06.05.2026 and a local time string, so GlsAdapter has to parse CET into UTC.

That is all adapter work. Once an event crosses into the core, it has one shape: carrier, tracking number, carrier status, normalized status, carrier timestamp, and a deterministic dedupKey.

The key is generated from four values:

export function generateDedupKey(input: DedupKeyInput): string {
  const carrier = normalizeCarrier(input.carrier);
  const trackingNumber = assertRequiredString(input.trackingNumber, "trackingNumber");
  const carrierStatus = assertRequiredString(input.carrierStatus, "carrierStatus");
  const carrierTimestampIso = normalizeTimestamp(input.carrierTimestamp);

  return [carrier, trackingNumber, carrierStatus, carrierTimestampIso].join(":");
}

That function is plain, but it carries the system. A carrier can send the same scan twice. A poll can retry after a timeout. A process can restart and fetch the same history again. If the fact is the same, the key is the same.

What surprised me was how much of the gateway became simpler once duplicate handling moved into the event identity instead of the polling loop. The poller does not need to remember what it saw last time. The adapters do not need per-carrier duplicate caches. The WebSocket layer does not decide whether something is new. The processor decides once, against PostgreSQL.

The processor is the load-bearing part

I was wrong at first about where the "real-time" complexity would live. I expected it to be WebSockets: connection cleanup, heartbeats, subscription maps, broadcast performance. Those were real problems, but they were not the hard correctness problem.

The hard part was making sure Redis never became the source of truth by accident.

createEventProcessor starts with a dedup lookup, then inserts into tracking_events with onConflictDoNothing(). That second guard matters because two workers can pass the lookup at the same time. The database still gets the final vote.

After the insert, it updates the shipment projection only when the new carrier timestamp is newer than last_event_at. Older events are still persisted. They just do not move the visible shipment state backwards.

The important shape is this:

const updatedRows = await options.db
  .update(shipments)
  .set({
    currentStatus: event.normalizedStatus,
    lastEventAt: event.carrierTimestamp,
    updatedAt: new Date()
  })
  .where(
    and(
      eq(shipments.id, event.shipmentId),
      or(isNull(shipments.lastEventAt), lt(shipments.lastEventAt, event.carrierTimestamp))
    )
  )
  .returning({ id: shipments.id });
projectionUpdated = updatedRows.length > 0;

That is the line between history and projection. An out-of-order scan belongs in history. It does not belong on the dashboard as the current state.

Then Redis publish happens after the database work. If the database write fails, the processor returns failed and does not publish. If Redis publish fails, the processor still returns processed with published: false. That asymmetry is deliberate. A WebSocket update can be missed. A carrier fact cannot be invented.

The integration tests capture both sides. One test inserts an invalid shipment id and asserts that Redis stays empty. Another makes the publisher throw redis unavailable and asserts the PostgreSQL event still exists. Those tests are more useful than a happy-path WebSocket demo because they pin down which failure the business can recover from.

Why not broadcast directly?

The simple version is tempting. A client subscribes to a tracking number. The processor receives an event. It loops over sockets and calls send().

That works until the process restarts, the send throws, or the event arrives in a worker that does not own the socket. Even in a single-process version, direct send ties event processing to live delivery. That is the dependency I wanted to avoid.

Redis Streams gave the gateway a narrow delivery contract. The processor writes one stream entry to tracking:events. The broadcaster consumes as group ws-broadcaster, parses the payload, looks up connection ids by tracking number, and sends JSON to sockets registered in the current process.

The broadcaster code has a small but telling rule: malformed stream entries are acknowledged.

At first that felt wrong. Acknowledging a bad message means admitting it will never be delivered. But leaving invalid JSON pending forever is worse. It blocks operational visibility and makes the group look unhealthy for a message that cannot succeed on retry. The test for that case writes { as the payload and asserts it gets logged and acked.

The other Redis detail is XAUTOCLAIM. If a consumer reads a stream entry and dies before acknowledging it, the message sits pending. The broadcaster reclaims stuck messages after 60 seconds. That is enough for the current single-service deployment, and it creates a clear upgrade path for multi-process delivery later.

The polling loop stays boring on purpose

The polling engine is less clever than the rest of the system. That is good.

It loads enabled carrier configs from the database, starts one loop per carrier, queries active shipments excluding delivered, returned, and deleted_at, then batches them by POLL_BATCH_SIZE. The default is 10.

Each shipment call runs through withExponentialBackoff. RateLimitError and CarrierError are retryable by default. The delay is baseDelayMs * 2 ** attempt, with the base coming from carrier_configs.backoff_base_ms.

The scheduler has one job: skip overlapping cycles. If a carrier cycle is still running when the next interval fires, it returns a structured skipped result with reason cycle_already_running. It does not start a second poller and hope the processor dedup saves the day.

That skip behavior matters because the PRD target is 100 active shipments across 3 carriers, with a carrier polling cycle under 30 seconds and WebSocket delivery under 200ms once the event enters the pipeline. If the poller stacks, the gateway creates its own load spike and every downstream metric lies.

The timestamp bug that exposed the database shape

The most useful gotcha came from pagination, not carrier integration. Cursor pagination repeated a shipment on page 2 even though the SQL condition compared the sort timestamp against the cursor timestamp.

The schema uses PostgreSQL timestamp without time zone. Passing a JavaScript Date through Drizzle and node-postgres introduced enough interpretation drift that the comparison did not behave like the stored value. The fix was to format the cursor as a PostgreSQL timestamp literal and cast it with ::timestamp.

That is why shipments.routes.ts has this helper:

function toPgTimestamp(value: Date): string {
  return value.toISOString().replace("T", " ").replace("Z", "");
}

It looks like formatting trivia. It is actually a pagination invariant. If the cursor repeats rows, clients may process the same shipment page twice or miss a page when they try to recover.

The part I left deliberately unfinished

The stats endpoint exposes a limitation instead of pretending the system has a metric it does not own yet. It returns active shipments and today's event counts by carrier, but error_rate and errors are null. The response includes a metrics_limitations entry that says carrier poll failure counters are not persisted yet.

That is not a polished answer, but it is the honest one. The polling engine logs retry attempts and per-cycle errors, and the build journal records the summary fields: carrier, shipments polled, events found, deduplicated events, and errors. Those numbers exist at runtime. They do not yet survive as queryable history.

I prefer that over a fake rate derived from current memory. A dashboard number that resets on restart is worse than no number because it invites operational decisions from incomplete evidence. If this gateway were being deployed for a client, the next increment would be a small poll-cycle table or metrics sink before anyone relied on /api/stats for carrier health.

The same honesty shows up in the success criteria. The system has the local loop, the production-start check, the Docker Compose smoke proof, and the mock-carrier journey. It does not have the 24-hour simulated-load soak. That missing box is not paperwork. It is the difference between "the architecture handles the failure model" and "this process survived a day of real runtime pressure."

The result

The finished local build passed 134 Vitest tests. The unit and integration coverage command passed at 83.32% statement and line coverage. The WebSocket E2E test publishes a Redis stream event and asserts delivery under 200ms after the broadcaster group exists. The mock-carrier demo registers DHL, DPD, and GLS shipments, subscribes over WebSocket, processes events through the real event processor, verifies duplicate suppression, and confirms the REST API shows in-transit shipments.

The 24-hour soak is still deferred. That matters. A gateway like this is not production-proven just because the happy path works for a few minutes.

But the failure model is in the right order. Carrier facts land in PostgreSQL first. Shipment state is a projection. Redis carries delivery. WebSockets are allowed to miss a push. The timeline is not allowed to lie.

Why I Made the Ledger Refuse Single Rows

Kingsley Onoh — Tue, 05 May 2026 19:35:17 +0000

The bug I kept designing around was not "wrong penalty amount."

Wrong amounts are visible. A supplier sees a credit note for EUR 8400, checks the contract, and pushes back. The operator can investigate. The trail exists.

The quieter failure is a penalty row with no mirror. One side of the system says the buyer is owed money. The other side never records the supplier-facing debit. The total looks correct in the dashboard because the credit row exists. The settlement builder can even pick it up. But the accounting story is incomplete, and it only becomes obvious when someone asks why the counterparty view does not match the buyer view.

That is the sort of error append-only systems can preserve forever.

So I made the ledger refuse single rows.

The system calculates supplier SLA penalties. A missed response target, a delivery window breach, a compounding daily penalty, a tier crossing, or a per-ticket miss becomes money owed. That money then needs to move through three phases: accrual, possible reversal, and settlement. Each phase has a tempting shortcut.

For accrual, the shortcut is one row with amount_cents and direction.

For reversal, the shortcut is updating the original row.

For settlement, the shortcut is marking the original row as settled.

All three shortcuts make the data easier to write and harder to defend.

The Shape That Forced the Design

The PRD had a hard constraint: penalty_ledger is append-only. Disputes, withdrawals, and corrections use compensating entries. Settlement membership lives outside the ledger. That sounds clean until the first application flow has to implement it.

The accrual worker receives a breach, loads the contract and clause, calls RulesEngine.calculatePenalty, and gets one answer back: no penalty, a domain error, an accrued amount, or a capped amount. The amount is not the ledger. It is only a financial fact waiting to become a record.

The ledger record has more obligations:

a credit side and mirror side
same amount and currency
same tenant, contract, counterparty, clause, and breach
same accrual period
same entry kind
same compensation reference when reversing

If any of those differ, the pair is not a pair. It is two rows that happen to be written near each other.

I originally thought the database trigger was the center of the solution. Block update and delete. Done. That part exists:

create trigger penalty_ledger_block_update
before update on penalty_ledger
for each row execute function block_penalty_ledger_mutation();

create trigger penalty_ledger_block_delete
before delete on penalty_ledger
for each row execute function block_penalty_ledger_mutation();

But that only stops mutation after insertion. It does not stop a bad insert. A database that blocks updates can still store a malformed truth forever.

That is where the F# domain layer earns its place.

The Load-Bearing Function

The most important code in the ledger path is not the insert statement. It is LedgerPair.create.

let private validate credit mirror =
    if Money.cents credit.Amount <= 0L || Money.cents mirror.Amount <= 0L then
        Error LedgerAmountMustBePositive
    elif
        credit.Direction <> LedgerDirection.CreditOwedToUs
        || mirror.Direction <> LedgerDirection.Mirror
    then
        Error LedgerPairDirectionInvalid
    elif credit.EntryKind <> mirror.EntryKind then
        Error LedgerPairKindInvalid
    elif credit.Amount <> mirror.Amount then
        Error(LedgerPairMismatch "amount must match")
    elif
        credit.AccrualPeriodStart <> mirror.AccrualPeriodStart
        || credit.AccrualPeriodEnd <> mirror.AccrualPeriodEnd
    then
        Error(LedgerPairMismatch "period must match")
    elif not (sameContext credit mirror) then
        Error(LedgerPairMismatch "tenant contract counterparty clause breach context must match")
    elif credit.CompensatesLedgerId <> mirror.CompensatesLedgerId then
        Error(LedgerPairMismatch "compensating ledger reference must match")
    elif credit.AccrualPeriodEnd < credit.AccrualPeriodStart then
        Error PeriodInvalid
    else
        Ok()

That function is deliberately narrow. It does not know about HTTP, Hangfire, PDF rendering, Invoice Recon, or even settlement grouping. It only knows what makes two candidate rows a valid accounting pair.

The private LedgerPair type matters. Callers cannot construct one directly. They can propose two LedgerEntryCandidate records, but only the domain module can expose the pair. That means the application layer does not get to "remember" to validate. It has no valid object unless validation has already happened.

This is the part I got wrong at first in my head: I thought append-only was mainly a persistence rule. It is not. It is a construction rule. Once bad ledger rows are inserted, append-only protects the bad rows just as strongly as the good ones.

Why The Rules Engine Stays Pure

The penalty math also stays outside the database. RulesEngine.calculatePenalty takes a PenaltyCalculationInput and returns a result. No connection string. No repository. No clock except the explicit AsOf value passed in.

That was not aesthetic. The engine needs deterministic recompute. Given the same contract, clause, breach, prior accruals, and timestamp, it should return the same penalty. The snapshot test covers twelve cases: flat penalties, capped penalties, monthly fee proration, tier crossing, overflow tier behavior, currency mismatch, daily compounding cap, missing units, inactive clauses, and pre-contract breaches.

The awkward case is previous accruals. Caps depend on what was already credited. Tiered penalties may need to accrue only the difference between the previous tier and the new one. Compounding daily penalties need to avoid adding the same days twice. So the rules engine receives PreviousAccruals, filters to credit-side rows for the same clause, and calculates incremental money from that prior state.

That looks like a database concern until it breaks. If the rules engine quietly queried the database itself, replay tests would become setup-heavy and timing-sensitive. By making prior state an input, the application layer owns retrieval and the domain layer owns the calculation.

Reversal Without Mutation

The reversal path is where the design either holds or collapses.

When a supplier disputes a breach and wins, the system cannot update the original accrual to zero. It cannot delete the row. It cannot mark the row as false and pretend the old financial position never existed.

It writes a reversal pair.

ReversalEngine.uncompensatedCreditAccruals first looks for accrual rows that do not already have a reversal pointing at them. Then reversalCandidate copies the amount, period, tenant, contract, counterparty, clause, and breach from the original accrual, changes the entry kind to Reversal, and sets CompensatesLedgerId.

That little filter is important. Without it, the same breach could be reversed twice. Append-only would preserve both reversals, and the system would show the supplier owed less than zero. The code does not rely on a user avoiding the button. It filters uncompensated credit rows and writes every reversal inside one transaction with the status change.

The design tradeoff is verbosity. A simple breach flow creates two rows on accrual. Accrual plus reversal creates four rows. A ledger explorer has to explain direction, entry kind, and compensation references. The UI has more work because the database refuses to simplify history for display.

I accept that. Accounting systems should make the audit easy and the write path strict.

Settlement Is Not a Ledger Mutation

Settlement introduced a second temptation: add settlement_id to penalty_ledger.

That would make queries simple. Find uncommitted rows where settlement_id is null. Mark them when the PDF is built. Done.

It would also puncture the ledger invariant. If building a settlement mutates a ledger row, the ledger is no longer an append-only record of penalty facts. It becomes a workflow table.

The repo uses settlement_ledger_entries instead. SettlementsRepository.listUncommittedAccruals selects credit-side accruals for the period, excludes rows that already have active settlement membership, and excludes rows with reversals. Then SettlementsRepository.insert writes the settlement row and membership rows in the same transaction.

That means settlement membership is append-like metadata around the ledger, not a change to the ledger event itself.

The cost is an extra table and a more careful query. The gain is that a settlement can be cancelled or released without rewriting what the penalty ledger said at the time of accrual.

The Part That Still Needs Pressure

The full suite passes: 12 domain tests, 6 data tests, 30 application tests, 10 API tests, and 3 UI tests. The dashboard load audit has tooling for 10000 breaches and 5000 settlements. The fake staging flow proves a Contract Lifecycle event can become an accrual, settlement, Invoice Recon outbox message, and settlement.posted hub event in the local harness.

But the PRD still has open success criteria around staged NATS-to-ledger time, Invoice Recon posting p95, and million-row ledger query behavior. That is the honest limit of the current proof. The invariants are strong. The operational envelope still needs live pressure.

What surprised me is how much of the system exists to protect against its own future convenience. Every obvious shortcut would make one screen or one query easier. Single-row ledger entries. Updating rows for reversals. Marking ledger rows as settled. Reading tenant identity live during PDF posting.

Each shortcut is fine until the first external audit, supplier dispute, or tenant rename.

The lesson I took from this build is narrow: append-only is not a database trigger. It is a system-wide refusal to let later workflow needs rewrite earlier financial facts.

Why I Refused to Re-Query the Tenant Row at Alert Dispatch Time

Kingsley Onoh — Sun, 26 Apr 2026 17:37:42 +0000

What does an audit-grade risk alert look like six months after the band crossing it describes, when the tenant has renamed itself twice in the interim, when the vendor has been merged into another, when the scoring rule has been retuned three times since? In a system that re-queries the source rows at dispatch time, the answer is whatever is true now. In this system, the answer is whatever was true at 14:03 on the Tuesday the alert was written.

The concrete shape of the problem: a risk alert is created at 14:03 Tuesday for a vendor crossing from medium to high. The Notification Hub is down for emergency Postgres maintenance. The dispatcher falls back to the failed-alert retry queue and tries again every 30 minutes. By Friday afternoon, when the Hub is back, the email body must still read Acme GmbH, the legal name that was current Tuesday, not the post-Wednesday rebrand to Acme Industrial GmbH that the operator entered through the settings UI on Wednesday morning. A live re-query of the tenants row at send time would emit the new name and reference an event at a company that didn't exist when it happened.

That's the constraint this whole architecture is designed around. Not the failure mode I worried about most. The one I knew would happen and didn't have a clean answer to.

The Real Problem

Most engineers I've talked to assume the issue is performance. They look at a table with a JSONB column called delivery_payload holding a fully-rendered tenant snapshot, and they ask why I'd denormalize. The tenants row has 11 identity columns. Why pickle them into JSON when I could just SELECT them again at dispatch time?

Performance has nothing to do with it. The dispatcher is fast either way. What matters is that an alert is a fact about what was true at a moment in time, and a re-query produces a fact about what is true now. Those two facts are not interchangeable. An auditor reading the alert ledger six months later wants to know what the operator saw on Tuesday afternoon, not what the system shows today.

Same problem on the report side. A vendor scorecard PDF generated in March needs to reprint byte-for-byte (or close enough to it) when an auditor downloads it again in September. If the underlying tenant or vendor row has been mutated in between, a live re-query produces a different document. The PDF that was approved by procurement leadership in March no longer exists on disk. It exists in the operator's email and nowhere else.

The mistake is one of category. The system was treating mutable rows as the source of truth for an immutable record. SQL was doing exactly what SQL does: returning the current value of the column. The error was in the architecture asking the question that way at all.

The Constraints

Three things made this hard.

Tenant identity is mutable by design. A CPO can update their legal name, registration number, address, and brand colors at any time through the settings UI. We don't lock fields after creation; that would be operationally hostile to the business reality of mergers and rebrands. So the row is the live identity, full stop.

Vendors get merged. When the operator confirms that two vendor_aliases rows point at the same legal entity, the system collapses them and the surviving vendor inherits the signals from the merged one. If an alert fires referring to "Vendor A" and that vendor gets merged into "Vendor B" three weeks later, a re-query at dispatch time would emit "Vendor B" in the email body. That isn't just confusing; it's wrong. The alert was about A.

Retries can run for days. The notification hub has a circuit breaker (5-failure rolling window, 60-second cooldown). If the Hub is genuinely unhealthy for a sustained period, the failed-alert retry job picks alerts back up every 30 minutes and tries again. There is no upper bound. I have seen alerts retry across 72-hour outages. Anything the dispatcher reads at send time can have moved arbitrarily far from where it was at create time.

The obvious moves were all bad. Locking the tenant row after first alert? Operationally hostile. Versioning every field on the tenant table with effective-date ranges? A second source of truth and a permanent maintenance burden. Re-creating the alert if the tenant changed? You can't, because the alert is a record of an event that already happened.

The Design

The architecture has two halves: storage immutability and rendering immutability. Both are necessary. Either alone is insufficient.

On the storage side, every alert carries a delivery_payload JSONB column populated at insertion. The capture path lives in lib/alerts/capture_payload.rb and is called from the alert dispatcher exactly once, at the moment the risk_alerts row is INSERTed:

# lib/alerts/capture_payload.rb
def call(vendor_score:)
  tenant = vendor_score.tenant
  vendor = vendor_score.vendor
  tenant_snapshot = Tenants::CaptureSnapshot.call(tenant.id)

  payload = {
    event_type: event_type,
    tenant: tenant_snapshot,
    vendor: vendor_block(vendor, tenant_snapshot),
    score: score_block(vendor_score, previous_composite, previous_band, direction),
    top_contributors: top_contributors_block(vendor_score),
    deep_links: deep_links_block(vendor),
    created_at: Time.now.utc.iso8601
  }

  deep_freeze(payload)
end

Tenants::CaptureSnapshot is the single canonical builder for the tenant identity block. Eleven columns from the tenants row plus a snapshot_at timestamp, returned as a frozen Hash. The shape is locked. Adding a column to tenants does not automatically add it to the snapshot. Every addition is a deliberate change to the snapshot, the templates that bind to it, and the test fixtures.

That's the first half. The dispatcher (Alerts::HubDispatchJob) then has exactly one rule: read alert.delivery_payload and never query tenants, vendors, or vendor_scores again. The job's class doc is a paragraph explaining this. The integration test fires an alert, mutates the underlying tenant row, then runs the dispatcher and asserts the emitted Hub event still contains the original literal values. Without that test, a casual refactor could re-introduce a Tenant.find(alert.tenant_id) call in the dispatcher, and the regression would only show up in production after the first sustained Hub outage.

The second half is in-memory immutability. A frozen top-level Hash doesn't prevent a careless caller from mutating a nested Hash:

# This still works on a top-level-frozen Hash:
payload[:tenant][:legal_name] = "Sneaky GmbH"
# raises FrozenError only if every nested level is also frozen

So the capture path walks the structure recursively:

def deep_freeze(obj)
  case obj
  when Hash
    obj.each_value { |v| deep_freeze(v) }
    obj.freeze
  when Array
    obj.each { |v| deep_freeze(v) }
    obj.freeze
  when String
    obj.freeze
  else
    obj
  end
end

A Sidekiq worker that loads the JSONB column gets a fresh Ruby Hash from JSON.parse, which is mutable, but the in-memory structure built by CapturePayload.call is physically immutable at every level the moment it leaves the method. That removes a class of bugs where a logging middleware or a serializer transformation accidentally mutates the snapshot in flight before it's emitted to the Hub.

The rendering side is the same idea applied to templates. Every Hub Liquid template registers with strict_variables: true. Every report ERB template uses a small helper called Reports::StrictFetch that walks dotted paths against the captured render context and raises StrictFetchError on any unresolved segment. A template that references {{ tenant.legal_nme }} (typo) doesn't silently emit an empty string. It raises during the test, before anything ships.

The CI gate that locks all of this is in test/integration/report_template_lint_test.rb. It renders every report template against captured render contexts for two distinct tenants, then parses each template for f("…") calls without a default: argument to enumerate the mandatory token set. The test will fail loudly if anyone adds a new template token without extending the snapshot, or adds a field to the snapshot but forgets to update a downstream template.

I had this turned into the cleverest part of the suite by accident. After the first version of the test passed, I noticed it would also pass if I weakened StrictFetch to silently return nil for missing paths. So I added a deliberate-failure regression test: a synthetic broken template that references {{ tenant.this_field_does_not_exist_anywhere }}. If anyone weakens the strict-fetch contract, that meta-test fails. Without it, the regression test for the regression test, the gate could rot.

What Surprised Me

The byte-identical re-render test for the four report types did not work the first time, and the failure mode taught me something I would not have figured out from documentation.

I picked the legal-footer line as my canary literal: Reg: HRB-123456 · Tax: DE987654321 · Berlin, DE. The PDF visually rendered correctly. But pdf-reader text extraction returned a nil for that line. After two hours of debugging, I figured out that wkhtmltopdf encodes the · HTML entity as a non-standard glyph escape that pdf-reader's content-stream decoder doesn't recognize, and the entire PDF text object containing that escape gets dropped from the extraction.

The fix was to pick canary literals from lines that don't contain · separators (header legal_name, address line1, contact email are all safe). But the lesson was the deeper one: byte-identical PDF re-rendering is impossible because wkhtmltopdf embeds creation timestamps and random object IDs in every render. CSV outputs are bytewise equal across renders; PDFs are content-equal via text extraction. I had to pick the right level of the immutability claim.

The Result

A 30-day audit-reprint test runs on every CI build. It captures a render context, generates the PDF and CSV, runs Timecop.travel(30.days), mutates the underlying tenants row, and re-renders. CSV output is exactly byte-equal. PDF text-extracted output contains the original tenant literals (Acme GmbH, Hauptstraße 10, procurement@acme-gmbh.example) and not the new ones. The same gate runs separately for the alert side: the dispatcher integration test fires an alert, mutates the tenant, runs the dispatcher, and asserts the emitted Hub event contains the pre-mutation legal name.

The architecture is denormalized. Every alert duplicates the tenant identity. Every report context duplicates the entire scoring snapshot. Storage cost is real but small (typical alert payload is ~3KB; a year of band-crossing alerts at 50 vendors per tenant is well under 100MB per tenant). The cost is paid once at write time and never paid again at read time.

The takeaway, if there is one: a snapshot has a different job from a normalized table. The normalized rows are operational state, useful for the live dashboard and the next score recompute, and entirely correct to mutate when the business changes. The frozen JSONB is the legal record. It's what you point at six months later when an auditor asks you to prove what the operator saw on a Tuesday afternoon in April.

The Per-Tenant Rate Limit That Wasn't Per-Tenant

Kingsley Onoh — Sun, 26 Apr 2026 17:33:43 +0000

Tenant A blocked correctly at request 11. Tenant B also blocked at request 11.

Phase 7 added a rate limit on POST /api/events that reads from tenants.config.rate_limits.events_per_minute. The default cap is 200 requests per minute. A tenant who needs more can be raised to 1000 via an admin PATCH. The integration test seeded two tenants (one on the default, one bumped to 100) and asserted that tenant A got blocked at request 11 while tenant B kept going. Tenant B was supposed to get nine more requests through.

The resolver function was fine. The unit tests on resolveTenantEventsRateLimit() all passed. Pass a tenant config with no override, get 200. Pass a tenant with events_per_minute: 100, get 100. Pass null, get 200. Pass anything over 1000, get clamped to 1000. Five tests, all green. The function did exactly what it was supposed to do.

The function was being called with null.

Where the Resolver Was Called With Null

@fastify/rate-limit accepts a max option that can be a function. The function receives the request and returns the cap to apply. Putting the resolver behind that callback was the design: each request asks "what is this tenant's limit?" at the moment the rate check runs, so changing a tenant's config takes effect immediately without a process restart.

For the resolver to do its job, it needs request.tenant already populated. That happens in authPlugin, which reads the X-API-Key header, looks up the tenant row, and attaches the tenant config to the request. The auth plugin runs in the onRequest lifecycle hook.

By default, @fastify/rate-limit also runs in onRequest. Fastify fires hooks in registration order within the same lifecycle stage. The rate-limit plugin was registered before the route was registered, and the route's auth runs as part of the route's plugin chain. The two onRequest hooks fired in an order that meant rate-limit's max callback ran first, with request.tenant still undefined.

The resolver, faithful to its contract, returned the default 200 for every request because every request looked like a tenant with no config override. Tenant A with the default and tenant B with 100 both got the same global cap. The endpoint behaved exactly as if the per-tenant feature didn't exist, but with no error and no warning, because both halves of the system were doing what they were told.

Why Moving Auth Wasn't an Option

The fix could not move auth. authPlugin runs in onRequest for every protected route in the entire API, and most of those routes do not need rate limiting. Pushing auth to preHandler to accommodate one route changes the lifecycle for every route, which moves a documented contract for one accidental dependency. That is the sort of fix that creates three new bugs to make one go away.

The fix also could not run rate-limit globally with a different hook. The rate-limit plugin can be registered globally with a custom hook stage, but that overrides the stage for all rate-limited routes. Every other rate-limited route in the codebase, including the admin endpoints with their own static caps, would be affected by a change made for events.routes.ts.

What was needed was a per-route override that moved only this rate check to a later hook stage, leaving the rest of the system on defaults.

hook: 'preHandler' Plus a Real keyGenerator

@fastify/rate-limit exposes a per-route option called hook inside the route's config.rateLimit object. Setting it to 'preHandler' means this specific route's rate check fires in the preHandler stage instead of onRequest. Auth has already run by then. request.tenant is populated. The resolver gets a real tenant config and returns the right cap.

config: {
  rateLimit: {
    hook: 'preHandler' as const,
    max: (request) => resolveTenantEventsRateLimit(request.tenant ?? null),
    keyGenerator: (request) => request.tenantId ?? request.ip ?? 'anonymous',
    timeWindow: '1 minute',
  },
},

Three things in that block matter, and the hook line is only one of them.

The keyGenerator is the second multi-tenant fix. The default key generator is the request IP. In a single-tenant deployment that is fine. In a multi-tenant deployment behind a shared NAT (two tenants whose servers happen to live in the same cloud region, two CI pipelines running tests behind the same egress address) the bucket gets shared. Tenant A burns through tenant B's allowance. The cap stops being per-tenant in a different way: the resolver returns the right number, but the bucket it counts against is wrong.

Setting keyGenerator to request.tenantId partitions the bucket per authenticated tenant. The fallback to request.ip covers the unauthenticated case (which the route does not allow, but defensive code is cheap and explicit defaults beat surprising ones).

The third thing is the resolver clamp at 1000. The admin route validates input at 1-1000, but a tenant whose config gets corrupted, or an old tenant from before the validation existed, could in principle have a number out of range stored in their JSONB config. The resolver caps the return value defensively. A misconfigured tenant cannot DoS the platform by setting events_per_minute: 999999 directly in the database.

Why a Single-Tenant Test Would Have Shipped This

The hook-ordering issue was not in the documentation for @fastify/rate-limit because it is not a bug in the plugin. The plugin defaults to onRequest because that is the right stage for rate limiting in 99% of cases. You want to reject over-limit requests as early as possible, before any work happens. The Hub's case is the 1% where the rate check depends on data that is only available after another hook has run. The plugin gives you the escape hatch, but you only know to use it once you have hit the problem.

The integration test that caught this was not a sophisticated test. It seeded two tenants, fired eleven requests for each, and asserted on the response codes. An earlier draft only seeded tenant A with a cap of 10 and asserted it got rate-limited at request 11. Tenant A was the only tenant. The test passed. I added tenant B with cap 100 specifically to verify that the per-tenant bucket worked, and the test failed with both tenants getting blocked.

If I had only tested one tenant, the rate limit would have shipped looking like it worked. Every tenant on the platform would silently be on the same global cap, and the only way to discover it would be a tenant raising a support ticket about being blocked at unexpected request counts. That is the kind of bug that runs in production for months because nothing visibly breaks.

The test fix was as simple as the production fix. Changing hook: 'preHandler' and keyGenerator: req.tenantId made the assertion pass. Total LOC change: two lines. Total time spent debugging from "why does tenant B also block" to "Fastify hook ordering": about an hour, most of it spent printing request.tenant from inside the max callback and confirming it was undefined.

What Else the Patch Quietly Got Right

H7 ships with per-tenant rate limiting that actually scopes per tenant. Tenant default is 200 requests per minute, raised to a configurable value via PATCH /api/admin/tenants/:id/rate-limit, capped defensively at 1000. The admin route uses spread-merge so updating rate_limits.events_per_minute preserves the rest of tenants.config (the channel credentials, the dedup window, the sandbox flag) without overwriting them. The PATCH was simple to write. Forgetting to use spread-merge would have been a different production incident: an admin update silently wiping every tenant's Resend API key.

Test coverage went beyond the spec. The resolver got five tests covering null config, missing override, explicit override, cap-at-1000, and the default. The integration suite got five more covering the per-tenant bucket, admin happy path, range validation at both ends, 404 on missing tenant, and config preservation across the update. The admin tests assert that channel credentials and dedup_window survive the PATCH untouched. The spread-merge contract is the thing that has to keep working, not just on the day it was written, but on every future change to the admin route.

The hook-ordering trick is now in the project's pattern 004 (per-route rate limit) as a note for the next route that needs a dynamic per-tenant cap. It will not be the last one. The same pattern will apply to per-tenant request limits on the suppressions endpoints, the templates endpoints, and any future endpoint where the cap depends on tenant config.

The fix is one line. The understanding it required is the lifecycle of every plugin in the request chain and the order in which Fastify will call them. That is the kind of detail that does not show up in a unit test for a resolver function.

The Cursor Pagination That Worked Until Five Rows Shared a Timestamp

Kingsley Onoh — Sun, 26 Apr 2026 17:27:55 +0000

It returned three.

The integration test had seeded five suppression rows in beforeAll, called GET /api/suppressions?limit=2, and walked the cursors forward expecting all five back. The first page returned two. The second page returned one. The third page came back empty.

The suppression list endpoint is a paginated GET. Tenants call it to see who they have manually blocked, who Resend marked as a hard bounce, and who complained about an email. The list grows over time and needs ordering by recency. Standard cursor pagination: order by (created_at DESC, id DESC), encode the last-seen tuple into a base64 cursor, and the next page asks for everything strictly less than that tuple. I had this pattern working on the notifications endpoint and wrote the suppressions version against the same template. Drizzle, Postgres, tuple comparison in SQL.

Where the Microseconds Disappeared

The seed used a single db.insert(...).values([row1, row2, row3, row4, row5]) call. Postgres performs that as one transaction. Every row gets the same clock_timestamp() for created_at, down to the microsecond Postgres records natively. From the database's point of view, the five rows are not just in the same second, not just in the same millisecond. They share a timestamp at six decimal places of precision.

The cursor encoding called row.createdAt.toISOString(). That returns ISO 8601 with millisecond precision. 2026-04-26T14:32:18.473Z. Nine digits in, the microsecond information is gone. When the next-page query compared the column value to the cursor value:

(created_at, id) < ('2026-04-26T14:32:18.473'::timestamp, 'last-id-here'::uuid)

Postgres compared the column's microsecond-precise value (14:32:18.473251) against the cursor's millisecond-precise value (14:32:18.473000). The column value was strictly greater. The tuple comparison returned false. Rows that should have appeared on the next page were filtered out as already-seen.

The first page returned the two newest rows correctly. The second page used the second row's millisecond-truncated timestamp as the cursor. The query asked for rows strictly less than that timestamp. The remaining three rows had timestamps strictly greater at the microsecond level, so they did not match. The endpoint silently returned an empty result. The test reported missing rows. Nothing in the logs hinted at why.

Why Notifications Never Hit This

The notifications endpoint had been running this pattern in production without a problem. The reason it worked is that production notifications are inserted one at a time, each one in its own transaction, each one with a distinct clock_timestamp(). Microsecond collisions are theoretically possible but practically vanishing.

Suppressions are different. The realistic write pattern is bulk: a tenant onboards and uploads their existing suppression list as a CSV, an admin runs a backfill from a previous notification provider, an auto-import grabs all hard bounces from the last 90 days. These all hit the database as batch inserts. Five rows sharing a microsecond stops being a contrived test fixture and becomes the normal case.

I considered three fixes. Round the column down to milliseconds at write time, so the cursor and the column always match. Switch the cursor format to a millisecond-precise serialization. Find a way to round-trip the microseconds losslessly through the cursor.

The first option destroys information. Postgres stores microseconds for a reason; throwing them away because the cursor is too coarse is a fix in the wrong layer. The second option amounts to the same thing. Whatever precision the cursor carries becomes the precision of the comparison, so reducing the cursor to milliseconds is identical to reducing the column.

The third option meant figuring out how to get microsecond precision out of Postgres in a form that survives a round trip through JSON, base64, and back into a SQL parameter.

to_char Saves the Round-Trip

Postgres has a function called to_char that formats timestamps using a pattern string. The pattern YYYY-MM-DD"T"HH24:MI:SS.US produces a string with microsecond precision in ISO-like format. Casting that string back to timestamp parses it without loss. The driver and the cursor never have to handle microseconds in JavaScript at all. The value moves as a string from query to cursor to next query, and Postgres does the precision work on both ends.

The list query selects an additional helper column with the formatted timestamp:

.select({
  id: tenantSuppressions.id,
  recipient: tenantSuppressions.recipient,
  reason: tenantSuppressions.reason,
  expiresAt: tenantSuppressions.expiresAt,
  createdAt: tenantSuppressions.createdAt,
  createdAtText: sqlOp<string>`to_char(${tenantSuppressions.createdAt} AT TIME ZONE 'UTC', 'YYYY-MM-DD"T"HH24:MI:SS.US')`.as('created_at_text'),
})

The createdAtText column is computed in the database for every returned row. The cursor encoder uses that string verbatim, paired with the row's id. The decoder pulls the string back out and casts it to timestamp in the next query's tuple comparison:

sqlOp`(${tenantSuppressions.createdAt}, ${tenantSuppressions.id}) < (${decoded.createdAtText}::timestamp, ${decoded.id}::uuid)`

The helper column is stripped from the response payload before the JSON goes back to the tenant, so the API surface stays clean. The byte-exact round trip happens entirely inside the cursor.

The fix is one extra column in the select, one cast in the where clause, one filter in the response mapper. Around fifteen lines of code. The behavior change is that bulk-inserted rows page correctly, even when five of them share a microsecond.

The Clever Fix That Failed Identically

An earlier attempt looked clever. Instead of relying on tuple comparison, I split it into two clauses: rows strictly older OR (rows at the same timestamp AND id strictly less). The Drizzle expression was something like or(lt(createdAt, cursorTs), and(eq(createdAt, cursorTs), lt(id, cursorId))). This is the textbook decomposition of a tuple comparison.

It also did not work, for the same reason. The eq(createdAt, cursorTs) branch compared the microsecond-precise column against the millisecond-truncated cursor, and the equality returned false. The second clause never matched. Same bug, more code.

I would not have noticed the OR-form failure if I had not been writing fresh tests against the seeded fixture. In production, with one row per transaction, both the tuple form and the OR form work. The seed was the only thing that exposed the precision mismatch. If I had written the suppressions endpoint with looser test fixtures (a setTimeout(10) between inserts, for instance) the bug would have shipped, and the first tenant to bulk-import a CSV would have hit it. Five hundred rows in, three hundred would page correctly and the rest would silently disappear from the list view.

The lesson I keep relearning is that test fixtures that simulate the realistic write pattern catch a class of bugs that loose fixtures hide. The seeded db.insert(...).values([...]) is exactly the shape of a bulk import. Writing the test that way was an accident, but it surfaced the bug at development time instead of in production.

Where the Recipe Lands

The fix shipped in batch 018, alongside the rest of the suppressions CRUD surface. The endpoint pages through bulk-inserted rows correctly regardless of how many share a timestamp. The cursor format is base64-encoded JSON containing the createdAtText string and the row id. The query plan uses the existing (tenant_id, created_at DESC, id DESC) index because tuple comparison maps cleanly to an index range scan in Postgres.

Pattern 006 in the project's pattern catalog already documented cursor pagination for the notifications endpoint. Rather than write a new pattern file, I extended 006 in place with the microsecond-precision recipe as a sub-section. The recipe is small enough that promoting it to a standalone pattern would be over-cataloging, but specific enough that the next endpoint that needs cursor pagination over bulk-insertable data can copy the exact to_char format string instead of rediscovering it.

Notifications are still on the millisecond-precision cursor. They have not hit the bug because nothing in the system bulk-inserts notifications. The day a backfill job lands, that endpoint gets the same fifteen-line treatment.

The whole thing is a two-character precision difference between two systems that both believe they are using ISO 8601, and the gap is invisible until five rows show up in the same transaction.

The Signing Bug You Cannot See Until a Tenant Tries to Verify You

Kingsley Onoh — Sun, 26 Apr 2026 17:27:51 +0000

A tenant registers a delivery callback URL. The Hub sends a POST whenever Resend reports an email bounced or was opened. The body is JSON. The header X-Hub-Signature carries an HMAC-SHA256 digest computed over that body using a secret that only the Hub and the tenant know. The tenant recomputes the digest on receipt and compares. If they match, the request is authentic. If they do not, the request is dropped.

This is the standard webhook-signing recipe. GitHub uses it. Slack uses it. Resend uses it on the way in. Now the Hub uses it on the way out.

The obvious implementation looks like this:

const body = JSON.stringify(event);
const digest = crypto.createHmac('sha256', secret).update(body).digest('hex');

It worked in tests. It worked in the integration suite. It worked against a mock callback server. Then I tried to write a verification snippet for the tenant docs in Python, and the digests did not match.

Why JSON.stringify Sinks Webhook Signing

JSON.stringify is not deterministic in any language. The order of keys in the output depends on the order they were inserted into the object. Two semantically identical objects can serialize to different bytes:

JSON.stringify({ a: 1, b: 2 })   // '{"a":1,"b":2}'
JSON.stringify({ b: 2, a: 1 })   // '{"b":2,"a":1}'

Both encode the same data. Both hash to a different digest under HMAC. A tenant who receives the callback, parses the JSON, then re-serializes it to compute their own signature, will produce different bytes than the Hub did, even when nothing about the payload changed.

This is the silent failure mode of webhook signing. Both sides are doing exactly what they were told. Both digests are correct for the bytes they were computed over. The bytes are different. The signatures mismatch. The tenant rejects the callback as a forgery, and there is no error message that explains why.

The Hub side controls the bytes that go on the wire. Whatever I sign, I send, and as long as the tenant hashes the raw bytes of what arrived, the digests agree. But "hash the raw bytes" is fragile. Express's body parser, FastAPI's request handler, any middleware in the chain that touches the body, will replace the original bytes with a re-stringified version. The tenant integration breaks not because of a bug in their code, but because of a default in the framework they used.

Three Things The Scheme Has To Do

I needed three things. The signing scheme had to be deterministic, so two encodings of the same data produced the same digest. It had to be specifiable, so a tenant could implement verification in any language without reading the Hub's source. And it had to be reusable, because Phase 7 added the delivery callback first but planned to extend the same signing pattern to suppression callbacks, generic webhook fan-out, and alert callbacks downstream.

The scheme I picked is canonical JSON: sort object keys recursively, build the JSON string by hand, hash that. Any tenant in any language that performs the same recursive sort produces the same bytes and the same digest. The contract becomes the canonicalization rule, not the byte stream.

Canonical JSON in Seventy Lines

The signing module is small. Three exported functions, one private helper, around 70 lines total in src/lib/outbound-signing.ts.

export function canonicalJson(value: unknown): string {
  if (value === null || typeof value !== 'object') {
    return JSON.stringify(value);
  }
  if (Array.isArray(value)) {
    return '[' + value.map((item) => canonicalJson(item)).join(',') + ']';
  }
  const keys = Object.keys(value as Record<string, unknown>).sort();
  const parts = keys.map((k) => {
    const v = (value as Record<string, unknown>)[k];
    return JSON.stringify(k) + ':' + canonicalJson(v);
  });
  return '{' + parts.join(',') + '}';
}

Three things to notice. Primitives delegate to JSON.stringify, which handles string escaping and number formatting correctly per the spec. Object keys get sorted lexicographically before serialization. The function is recursive and uses no whitespace, so the output is byte-identical to what a JCS-style canonicalizer in Python or Go would produce given the same input.

What the function deliberately does not handle: Date, Map, Set, BigInt. None of those round-trip through JSON cleanly. Outbound callback payloads are intentionally pure JSON values, type-checked at the call site. Refusing to support exotic types keeps the canonicalization rule auditable in one paragraph rather than a footnote.

The signing function builds on top:

export function buildSignedOutboundRequest(
  event: unknown,
  secret: string,
): { body: string; signatureHeader: string } {
  const body = canonicalJson(event);
  const digest = crypto.createHmac('sha256', secret).update(body).digest('hex');
  return { body, signatureHeader: `sha256=${digest}` };
}

The body and the signature are returned together. This is the design decision that took the longest to reach. The first version returned them separately and let the caller choose what to send over the wire. That created room for a class of bugs where the caller signed one set of bytes and sent a different set: log the body, then send JSON.stringify(event), sign with canonicalJson(event), and ship the wrong thing. Returning them as a pair means the bytes that get hashed are the bytes that get transmitted. The caller cannot accidentally split them.

The sha256= prefix on the header value follows the convention GitHub and Slack use. It exists so that the scheme can later add SHA-512 or another algorithm without breaking parsers, and so tenants writing verification code can split on = and validate the algorithm name first.

The Test Story That Inverted

I assumed the testing story would be the hard part. Cryptographic functions resist mocking. You cannot mock HMAC into returning a fixed digest, because the whole point is determinism over real input. So I expected a lot of test infrastructure: helper builders, fixture payloads, golden digests checked into the repo.

The opposite happened. Because the function is pure and deterministic, the tests are five-line affairs. Pass an input. Assert on the output digest. No mocks. No setup. The HMAC determinism tests run in microseconds:

test('signs canonically: key order does not affect digest', () => {
  const secret = 'test-secret';
  const a = signOutboundPayload({ a: 1, b: 2 }, secret);
  const b = signOutboundPayload({ b: 2, a: 1 }, secret);
  expect(a).toBe(b);
});

That single test catches the entire class of bug that motivated the canonicalization in the first place. If JSON.stringify ever sneaks back into the signing path, this test fails immediately. It catches regression in any future refactor.

The harder testing problem turned out to be on the dispatch side, not the signing side. The route that triggers a callback uses void dispatchDeliveryCallback(...).catch(...). That is fire-and-forget: the webhook handler can return 200 to Resend without waiting for the tenant's callback URL to respond. Production behavior is correct: the Hub never blocks Resend on a slow or down tenant. But naive integration tests race against the dispatch. The test calls app.inject(), gets a 200 response, then asserts on the database row that should have been updated by the callback handler. The callback might still be in flight.

Three options for testing this. Option one: add a test-only hook that returns the in-flight promise so the test can await it. Option two: use setImmediate() and poll. Option three: keep the production fire-and-forget contract untouched and use a polling helper in the test that waits up to two seconds for the assertion to become true.

I picked option three. The production behavior is the contract. Adding a test-only side channel means production code carries a hook that exists only for tests, which inverts the relationship: the test is supposed to verify the production code, not constrain it. The polling helper is uglier but bounded: typical cases resolve in 10 to 50 milliseconds because both the mock callback server and the database are local, and the upper bound of two seconds catches a real hang without hanging the test suite forever.

What Ships in Phase 7

The signing module ships in batch 011 of Phase 7. The webhook integration in batch 012 wires the dispatcher to call it on every Resend delivery event. The end-to-end test in batch 013 fires a mock Resend webhook into the Hub, watches the database row get inserted, then watches a mock callback server receive a signed POST with the right digest. All three batches add 14 new tests on top of the existing 309. None of them mock HMAC.

The tenant verification snippet in USER_SETUP.md Section 10 is intentionally explicit about reading raw request bytes. In Node, express.raw({ type: 'application/json' }) instead of express.json(). In Python, request.get_data() instead of request.json(). The snippet works because the Hub signs canonical JSON and sends those exact bytes. A tenant who hashes the raw body bytes verifies correctly. A tenant who reparses and re-serializes does not, and the snippet says so up front.

Phase 7 7b extracted canonicalJson and signOutboundPayload from the delivery-callback module into src/lib/outbound-signing.ts so the next callback family (suppression notifications, generic webhook fan-out, alert callbacks) uses the same scheme without copy-paste. The canonicalization rule is now load-bearing for every outbound HTTP call the Hub will ever make.

The signing module is 73 lines. The bug it prevents took an afternoon to find and would have taken a tenant a week to debug on their side.

Stripe Said Past Due. The State Machine Said No. Both Were Right.

Kingsley Onoh — Sat, 18 Apr 2026 11:25:43 +0000

A paused subscription receives a customer.subscription.updated webhook from Stripe. The payload says the new status is past_due. The local state machine has eight states and fifteen valid transitions, defined as a single Elixir map literal. paused can transition to active. That is the only outbound edge. There is no paused to past_due entry. The state machine says this transition doesn't exist.

What do you do with the event?

Rejecting the event is the safe choice. Invalid transition, log a warning, drop the payload. Clean, principled, and wrong. Stripe doesn't send a status change in isolation. The same webhook payload contains updated period dates, metadata changes, and potentially a new plan assignment. Rejecting the event to protect the status field means losing everything else in the payload.

Forcing the transition is the pragmatic choice. Override the state machine, accept whatever Stripe says, move on. Also wrong. The state machine exists because downstream business logic depends on it. The dunning engine creates retry sequences when a subscription enters past_due. If a paused subscription suddenly appears as past_due through a forced transition, the dunning engine starts chasing a payment that was intentionally paused. The customer gets escalating "your payment failed" notifications for a subscription they paused on purpose.

But the invalid transition is a symptom, not the disease. Stripe and the local data model can disagree about the current state of a subscription, and neither is wrong. Stripe is the source of truth for payment processing. The local state machine is the source of truth for business logic execution. When they conflict, you need an architecture that handles the disagreement without losing data or breaking invariants.

Two Sources of Truth, Zero Arbitration

This isn't a sync problem. Sync implies one system is behind and needs to catch up. This is a disagreement: Stripe says the state is X, the local model says the transition to X is illegal, and both positions are defensible.

Stripe can put a subscription into states the local model doesn't allow because Stripe's state machine is broader and serves a different purpose. Stripe tracks billing states across all possible payment scenarios, including edge cases around payment method updates, invoice finalization timing, and subscription schedule modifications. The local model tracks lifecycle states for business logic: when to start dunning, when to compute churn, when to notify the customer. These are overlapping but non-identical concerns.

The moment I accepted that these two state machines would diverge, the design became clear. The system needed to handle three cases: agreement (both say the same thing, proceed normally), valid disagreement (the transition is in the map, accept it), and invalid disagreement (the transition is not in the map, accept the data but not the transition).

The Strip, Don't Reject Pattern

The core of this design lives in a single function: maybe_strip_status/3 in the SubscriptionProcessor module.

defp maybe_strip_status(stripe_data, nil, _new_status), do: stripe_data
defp maybe_strip_status(stripe_data, prev, new) when prev == new, do: stripe_data

defp maybe_strip_status(stripe_data, previous_status, new_status) do
  case StateMachine.transition!(previous_status, new_status) do
    :ok ->
      stripe_data

    {:error, :invalid_transition} ->
      Logger.warning(
        "SubscriptionProcessor: invalid transition #{previous_status} -> #{new_status}, " <>
          "keeping current status"
      )

      Map.put(stripe_data, "status", previous_status)
  end
end

When the state machine rejects a transition, the function does not reject the event. It replaces the incoming status with the previous status and returns the rest of the payload untouched. Period dates, trial dates, metadata, plan references: all of it passes through. Only the illegal status change gets stripped.

The first two guard clauses handle the cold-start cases. When previous_status is nil (brand new subscription, no prior state to compare), accept whatever Stripe sends. When the previous and new statuses are identical (Stripe sent an update that didn't change the status), pass through without checking the transition map.

This is a deliberate tradeoff. The local status can drift from Stripe's status. A subscription might be past_due in Stripe's records and paused in the local database. I accepted this inconsistency because the local model controls what actually happens: which dunning sequences fire, which notifications send, which metrics count the subscription as churned. If Stripe's status is wrong from the local model's perspective, the local model wins for execution and Stripe wins for billing. Nobody is the universal authority.

Idempotency Under Disagreement

The state disagreement compounds with another problem: event ordering. Stripe doesn't guarantee webhook delivery order. A customer.subscription.updated event from 3:00 PM can arrive after one from 3:05 PM. If the 3:05 PM event was already processed and moved the subscription to active, the 3:00 PM event now carries a stale status that the state machine might reject.

The idempotency model handles this with a three-state check, not the typical two-state (seen/unseen) approach. Every event is stored with a composite key: tenant_id:stripe_event_id. Before processing, the system checks:

New: No record exists. Process normally.
Duplicate: A record exists with processed_at set. Skip entirely, return success.
Processing: A record exists without processed_at. Another Oban worker is handling this event right now. Skip to avoid double-processing.

The third state is the one most implementations miss. Without it, two workers pulling the same event from the Oban queue would both attempt to process it, and both would try to create downstream records (dunning attempts, notifications). The database-level unique constraint on the idempotency key catches this at the persistence layer, but the three-state check catches it earlier and avoids the exception entirely.

The Dunning Guard Nobody Asks About

The paused-to-past-due edge case has a second layer of protection deeper in the processing pipeline. Even if the status stripping somehow failed and a paused subscription appeared as past_due, the dunning trigger has its own guard:

if prev_status == "paused" do
  Logger.warning(
    "SubscriptionProcessor: paused subscription transitioned to past_due, " <>
      "skipping dunning for sub #{subscription.id}"
  )
end

This is belt-and-suspenders engineering for a specific Stripe behavior I discovered during development. Stripe can emit a past_due status for a paused subscription when a pending invoice from before the pause fails to collect. The subscription was paused intentionally, but Stripe's billing engine still tries to finalize the outstanding invoice. If it fails, Stripe marks the subscription past_due even though the customer explicitly paused it.

Without this guard, the dunning engine would start a 7-day escalation sequence (email at day 1, email at day 3, Telegram at day 5, both channels at day 7, then automatic cancellation) for a subscription the customer paused. The customer experience would be: "I paused my subscription. Three days later I got a Telegram message saying my payment failed and I need to act immediately." That is not a recoverable customer relationship.

What I Would Redesign

The status stripping approach works, but it has a gap: the system does not reconcile. If the local status diverges from Stripe's status, it stays diverged until the next valid transition arrives. A reconciliation job that periodically fetches subscription status from the Stripe API and compares it against local state would close this gap. I didn't build it because the current system is event-driven by design (no polling), and adding a reconciliation poller would create a second source of state mutations that the webhook pipeline doesn't expect.

The churn calculator has a related bootstrap problem. It computes churn rate as churned_count / active_count_at_period_start, where the denominator comes from the previous day's metrics snapshot. On the first day of operation, there is no previous snapshot. The first churn rate is always 0.0000, regardless of what actually happened. I accepted this because building temporal query infrastructure to count active subscriptions at an arbitrary past timestamp added complexity that the 99.9% case (day 2 and beyond) doesn't need. But the first day's metrics are wrong, and there is no warning in the dashboard about it.

The Lesson Is About Modeling, Not Stripe

Stripe is the specific case, but the pattern applies to any system that processes events from an external authority. Payment providers, shipping APIs, identity providers, government reporting systems: they all emit state changes according to their own transition rules. Your internal model is a subset of their model, tuned for your business logic, and the two will disagree.

The instinct is to treat the external system as the single source of truth. Let Stripe win every argument. The problem is that "letting Stripe win" means your business logic executes against a state model you do not control and cannot predict. The alternative isn't to fight the external system. It is to separate the concerns: accept their data, enforce your transitions, log the disagreements, and build guards at every point where the disagreement could trigger unintended side effects.

Build for disagreement, not consensus. The external world does not know your rules, and it does not care.

Why Anomaly Detection Can't Block the Ingestion Pipeline

Kingsley Onoh — Sat, 18 Apr 2026 11:23:23 +0000

The first version of the anomaly evaluator ran inline. A batch of 100 readings flushed to TimescaleDB, then the same function called evaluate_batch, which fetched alert rules, computed rolling statistics from the continuous aggregates, checked cooldown windows, persisted alerts, published NATS events, and fired HTTP calls to the Notification Hub and Workflow Engine. All of it synchronous. All of it in the same call stack as the NATS consumer loop.

It worked for about thirty seconds. Then the Workflow Engine returned a 502, the reqwest client waited for its default timeout, the batch flush stalled, NATS messages piled up, and the consumer loop fell behind by 3,000 messages before I killed the process.

The lesson was obvious in retrospect: a sensor data pipeline that sustains 5,000 readings per second cannot wait for a downstream HTTP response. The interesting question was what "don't wait" actually means at the architecture level. Making the HTTP call async solves the timeout. It doesn't solve the failure model. I needed every component downstream of the batch insert to be able to fail, silently, without the consumer loop noticing or caring.

The pipeline, end to end

A sensor message arrives on NATS as JSON. The consumer deserializes it, validates the value (must be finite, metric must be non-empty), resolves the tenant via the TenantResolver's DashMap cache, auto-registers the device if unknown, and pushes the reading into the BatchBuffer. When the buffer hits 100 readings or 500 milliseconds elapse, whichever comes first, it flushes.

The flush itself is the only synchronous bottleneck I allow. The execute_insert method builds UNNEST arrays for all seven columns and sends a single INSERT to TimescaleDB. If that fails, it retries once after 100 milliseconds. If the retry fails, it logs at error level and drops the batch. There's no infinite retry loop. No dead-letter queue. The batch is gone.

I made that choice deliberately. At 5,000 readings per second, a retry queue that grows faster than it drains is worse than data loss. Thirty seconds of buffered retries at full throughput means 150,000 queued readings. The server runs in 512MB. The math doesn't work. Drop the batch, log the failure, let the operator investigate. The continuous aggregates will smooth over a missing 100-reading gap, and the anomaly detector uses rolling statistics over minutes, not individual readings.

The buffer swap pattern

The BatchBuffer holds a Vec<ValidatedReading> behind a tokio::sync::Mutex. The critical design detail is in the push method: when the buffer reaches capacity, the code calls std::mem::replace to swap the full Vec with a fresh one, then drops the Mutex guard before executing the INSERT.

let readings = std::mem::replace(&mut *buf, Vec::with_capacity(self.batch_size));
drop(buf); // release lock before DB I/O
let flushed = self.flush_to_db_with_retry(readings).await?;

This matters because the INSERT takes milliseconds. If the lock were held during the INSERT, every concurrent push call would block. At 5,000 messages per second, even a 5ms INSERT means 25 messages queued behind the lock. The swap pattern reduces the critical section to a memory operation, not a network round-trip.

I considered using a lock-free structure like a crossbeam channel or a ring buffer. The Mutex won the trade-off because the batch logic needs to check the current buffer length before deciding to flush, and that check-then-act pattern doesn't map cleanly to a channel. The swap keeps the lock hold time under a microsecond, which is well below the threshold where contention would show up at 5,000 messages per second.

Detection runs post-flush, not inline

After the BatchBuffer returns the flushed readings, the consumer calls run_anomaly_evaluation. This function is the firewall between ingestion and detection. Here's what it looks like:

async fn run_anomaly_evaluation(
    pool: &PgPool,
    nats_client: &async_nats::Client,
    readings: &[ValidatedReading],
    notification_emitter: &NotificationEmitter,
    workflow_trigger: &WorkflowTrigger,
) {
    match evaluate_batch(pool, nats_client, readings, notification_emitter, workflow_trigger).await {
        Ok(alert_ids) => {
            if !alert_ids.is_empty() {
                info!(count = alert_ids.len(), "Anomaly evaluation created alerts");
            }
        }
        Err(e) => {
            warn!(error = %e, "Anomaly evaluation failed, continuing");
        }
    }
}

The return type is (), not Result. Errors are logged at warn level and swallowed. The consumer loop continues processing the next NATS message regardless of whether detection succeeded. A database timeout in the rule query, a serialization failure in the NATS publish, a network error calling the Notification Hub: none of these stop ingestion.

Deduplication before evaluation

The evaluate_batch function doesn't evaluate every reading in the batch. It extracts unique (tenant_id, device_id, metric) tuples, keeping only the latest value for each. A batch of 100 readings from 5 devices reporting 3 metrics each produces 15 unique tuples, not 100. The deduplication uses a HashSet<ReadingKey> and iterates the batch in reverse so the most recent reading for each tuple wins.

This was a performance decision. Fetching alert rules from PostgreSQL and computing rolling deviation statistics from the readings_1m aggregate table costs a query per unique tuple. At 100 queries per batch, the evaluator would be the bottleneck. At 15 queries per batch, it completes in single-digit milliseconds.

Three condition types, one evaluation path

Each alert rule specifies a condition type: above_threshold, below_threshold, or deviation_from_mean. The first two are trivial comparisons. The third runs a SQL query against the readings_1m continuous aggregate, computing avg(avg_value) and stddev(avg_value) over the configured window_minutes. If fewer than 5 buckets exist in the window, the check skips entirely.

That minimum sample count of 5 was the result of a frustrating cold-start problem. When a new device starts reporting, the first few readings have no aggregate history. A threshold of 2 standard deviations from a mean computed from 2 data points fires on everything. The system was generating hundreds of false alerts during device onboarding. Setting the floor at 5 samples eliminated the noise without significantly delaying real anomaly detection. Five minutes of readings at one-per-second gives 5 one-minute aggregate buckets. That's the minimum window before deviation detection activates.

Cooldown prevents alert storms

Without cooldown, a sensor stuck at a high value generates an alert on every batch flush. At 500ms intervals, that's 120 alerts per minute for a single sensor. The check_cooldown function queries the alerts table for any alert with the same rule_id and device_id created within the last N minutes (default 15). If one exists, the alert is suppressed and logged at info level.

The cooldown query hits an index on (rule_id, device_id, created_at). I briefly considered an in-memory cooldown cache (DashMap with TTL, similar to the TenantResolver), but the database-backed approach won because cooldown state needs to survive process restarts. If the service crashes and restarts, a memory-based cooldown would fire duplicate alerts for every rule that was in cooldown before the crash.

Fire-and-forget for ecosystem calls

The Notification Hub and Workflow Engine integrations use the same pattern: serialize the payload, spawn a Tokio task, return immediately. The spawned task makes the HTTP call and logs the result. The caller never sees the response.

tokio::spawn(async move {
    let result = client.post(&url).header("X-API-Key", &api_key).json(&event).send().await;
    match result {
        Ok(resp) => { /* log success or failure status */ }
        Err(e) => { warn!(error = %e, "Failed to send event"); }
    }
});

The Workflow Engine call adds HMAC-SHA256 signing. The compute_signature method takes the secret and raw body bytes, produces a sha256=<hex> signature, and attaches it as X-Hub-Signature-256. If no secret is configured, the header is omitted entirely.

This is at-most-once delivery by design. If the Notification Hub is down, the alert event is lost. I chose this over at-least-once (which would require a persistent outbox or retry queue) because the alerts are persisted in the database regardless. The Notification Hub is a convenience layer for pushing alerts to email or Telegram. An operator who checks the API's /api/alerts endpoint will always see the full alert history, even if the Hub missed a notification.

What I'd redesign

The anomaly evaluator currently runs on the same Tokio runtime as the NATS consumer and the Axum HTTP server. At 5,000 readings per second this is fine. At 50,000, the rule evaluation queries would compete with API queries for the 20-connection database pool. I'd split the evaluator into a separate process that consumes a NATS subject of "batches flushed" events, running its own connection pool. The ingestion binary would publish a summary event after each successful flush instead of calling the evaluator directly.

The other weak point is the fire-and-forget pattern. At scale, lost notifications become a support problem. The fix is an outbox table: persist the notification payload alongside the alert, then have a background worker poll the outbox and deliver with retries. The outbox adds latency and complexity, but it makes notification delivery observable. Right now, the only way to know a notification was lost is to notice a gap in the Notification Hub's event log. That's not good enough for a production system where the person on call needs to trust that critical alerts reach them.

I Spent a Week Securing Webhook Ingestion. The Real Attack Surface Was Delivery.

Kingsley Onoh — Sat, 18 Apr 2026 11:20:14 +0000

I ran the security review two weeks after the first deployment. The ingestion side looked solid: HMAC signature verification using crypto.timingSafeEqual, rate limiting at 1,000 requests per minute, payload size capped at 1MB, idempotency deduplication on every incoming event. I was satisfied with the input boundary. Then I traced what happens after an event is accepted, through the delivery worker and out to destination URLs, and realized I'd spent a week protecting the wrong end of the system.

The ingestion endpoint validates who is sending. But the delivery worker, the component that forwards payloads to downstream URLs registered by tenants, makes outbound HTTP requests from inside the server's network. That side had no protection at all.

Two attack surfaces, nothing in common

A webhook gateway has two distinct threat models, and they require completely different defenses.

On the ingestion side, the threat is an external attacker sending malicious payloads: forged signatures, oversized bodies designed to exhaust memory, replay attacks using captured requests, and deeply nested JSON meant to overflow the parser stack. The defense strategy is conventional. Validate everything at the boundary before it touches the database.

On the delivery side, the threat comes from inside. A tenant registers a destination URL. The URL passed validation when it was created: api.customer.com resolved to 34.120.18.42, a legitimate GCP load balancer. But URLs are just DNS pointers, and DNS records change. Three weeks later, the same hostname resolves to 169.254.169.254, the cloud metadata endpoint on every major provider. The webhook gateway, running inside the VPS, dutifully POSTs the event payload to the internal metadata service.

This is Server-Side Request Forgery through DNS rebinding. The gateway becomes an open proxy for any tenant who controls their DNS records. And the creation-time URL validation caught none of it, because the DNS resolution happened at delivery time, weeks after the destination was registered.

The exposure isn't limited to metadata endpoints. Private IP ranges (10.0.0.0/8, 172.16.0.0/12, 192.168.0.0/16), loopback addresses, the link-local range (169.254.0.0/16), IPv6 unique local addresses (fc00::/7), and protocol handlers like file:// or gopher:// are all reachable from the delivery worker's network context. The attack surface is larger than it first appears.

What made it hard

Three constraints shaped the fix.

First, the system delivers webhooks continuously. The SSRF check runs on every delivery, not just at destination creation. Any latency added to the DNS resolution and IP validation eats into the 10,000ms delivery timeout budget. The check needed to complete in single-digit milliseconds under normal conditions, which ruled out external SSRF-detection services and pre-resolution caching (stale DNS entries defeat the purpose).

Second, the existing delivery worker was the hottest code path in the system: load event, load destination, make HTTP call, record result, calculate backoff, schedule retry or promote to dead letter. Ten concurrent workers execute this pipeline on every job. Inserting validation in the middle of this sequence meant touching every execution path without breaking the retry logic, the exponential backoff calculations in calculateBackoffMs(), or the dead-letter promotion threshold.

Third, some destinations legitimately resolve to IPs that look suspicious to a naive blocklist. A customer running their webhook handler on a small hosting provider might have an IP range that borders private space. The validation needed to be strict about RFC 1918 ranges while returning clear, actionable error messages when it rejected a delivery, so tenants could debug the issue without opening a support ticket.

Six layers, added iteratively

The initial deployment shipped with HMAC signature verification and rate limiting on the ingestion endpoint. The remaining layers arrived in a separate wave of commits, all prefixed fix( rather than feat(, because each one addressed a gap I discovered after the pipeline was already handling traffic.

Ingestion input protection:

Rate limiting caps ingestion at 1,000 requests per minute per source. Payload size is limited to 1MB. JSON parsing enforces a nesting depth limit of 20 levels (MAX_JSON_DEPTH in src/server.ts) to prevent stack overflow from recursive structures. HMAC signature verification uses crypto.timingSafeEqual to prevent timing side-channels. The verifySignature() function in src/ingestion/signature.ts supports both hmac-sha256 and hmac-sha1, strips algorithm prefixes (GitHub sends sha256=<hex>, Stripe uses t=<unix>,v1=<hex>), and applies a configurable timestamp tolerance. The extractTimestampFromHeader() function parses Stripe's format specifically: t=<unix_seconds>, at the beginning of the header value, converted to milliseconds. Any signature older than 5 minutes (the SIGNATURE_TOLERANCE_MS default of 300,000ms) is rejected before the HMAC is even computed.

Delivery output protection:

Layer one is validateDestinationUrl() in src/lib/url-validator.ts, called at destination creation. It rejects non-HTTP protocols immediately. It blocks localhost and checks raw IP addresses against private ranges via isPrivateIp(). The IPv4 check walks the octets: 127.x is loopback, 10.x is private, 172.16-31.x is private, 192.168.x is private, 169.254.x is the link-local range that covers cloud metadata endpoints on AWS and GCP. IPv6 gets separate handling: ::1 (loopback) and fc00::/7 (unique local, covering both fc and fd prefixes).

Layer two is resolveAndValidateUrl(), called before every delivery. This is the DNS rebinding defense:

// src/lib/url-validator.ts
export async function resolveAndValidateUrl(url: string): Promise<string> {
  validateDestinationUrl(url);

  const parsed = new URL(url);
  const hostname = parsed.hostname.replace(/^\[|\]$/g, "");
  if (net.isIP(hostname)) return url;

  const addresses = await dns.resolve4(hostname).catch(() => []);
  const addresses6 = await dns.resolve6(hostname).catch(() => []);
  const allAddresses = [...addresses, ...addresses6];

  if (allAddresses.length === 0) return url;

  for (const addr of allAddresses) {
    if (isPrivateIp(addr)) {
      throw new UnsafeUrlError(
        `DNS rebinding detected: '${hostname}' resolves to private IP '${addr}'`
      );
    }
  }
  return url;
}

Every A and AAAA record is checked. If any resolved IP falls in a private range, the delivery is rejected with a specific error message naming the hostname and the offending address. The function resolves IPv4 and IPv6 in parallel to minimize latency. If DNS resolution fails entirely, the request is allowed through because the subsequent fetch will fail with a network error on its own. No errors are swallowed silently.

I got this wrong the first time. The initial implementation only had layer one: creation-time validation. I assumed that if the URL was safe when the tenant registered it, it would stay safe. It took reading through SSRF post-mortems to realize that DNS-based attacks bypass creation-time checks entirely. The fix shipped as its own commit: fix(lib): add SSRF protection, helmet headers, raw body HMAC, patch drizzle CVE.

The delivery worker also caps redirect chains at 3 hops. The deliverWebhook() function in src/delivery/http-client.ts follows redirects manually instead of relying on fetch's default behavior, and truncates response bodies to 4,096 bytes to prevent a destination from returning a multi-megabyte response that fills the deliveries table.

Data protection:

Header sanitization strips Authorization and Cookie headers from stored payloads at ingestion time. These headers might be present in the original webhook request (some providers include auth tokens), and forwarding them to a third-party destination would leak credentials. Signing secrets for HMAC verification are encrypted at rest with AES-256-GCM. The encrypt() function in src/lib/crypto.ts generates a random 12-byte IV per encryption and packs IV, auth tag, and ciphertext into a base64 string. The decryption key lives in an environment variable (SIGNING_SECRET_KEY), separate from the database. A breach that exposes the sources table gives the attacker ciphertext, not usable keys.

What changed

The security hardening added roughly 400 lines of validation code to a system that was already functionally complete. The url-validator.ts module alone is 195 lines of checks that don't change what the system does. They change what it refuses to do.

Before: the delivery worker would POST to any URL a tenant registered, follow unlimited redirects, and forward all original headers. After: every delivery passes through protocol filtering, hostname blocking, static IP range checks, dynamic DNS resolution with IP validation, redirect limiting at 3 hops, header sanitization, and response body truncation at 4,096 bytes.

The ingestion side now has signature verification with timing-safe comparison and 5-minute stale tolerance, payload caps at 1MB, and JSON depth limiting at 20 levels. Six overlapping layers, three on each side of the persist-before-process boundary.

None of these layers trust the previous one. The delivery-time DNS check doesn't assume the creation-time URL check caught everything. The JSON depth limit doesn't assume the payload size limit prevented pathological inputs. Every layer operates under the assumption that the one before it was bypassed or insufficient. When you build infrastructure that accepts input from the internet and acts on it inside your network, the input validation is the obvious problem. The delivery side, where your system becomes a network actor on behalf of untrusted tenants, is where the real exposure hides.

Building a DAG Orchestrator That Rewrites Its Own Execution Plan

Kingsley Onoh — Sat, 18 Apr 2026 11:16:18 +0000

Step 7 is a condition: if the HTTP response came back 200, continue to step 8 and step 9. If not, skip them and jump to step 10. Five steps downstream of a single boolean, two possible paths, and a directed acyclic graph that only understands one thing: which step depends on which.

DAGs don't branch. That was the problem I had to solve.

The linear model

The linear step types worked without issues. HTTP calls, data transforms, delays, sub-workflows. The orchestrator in orchestrator.py called topological_sort() from graph.py, got back a flat list of steps in dependency order, and executed them one by one. After each step completed, its output merged into a shared JSONB context accessible to every downstream step via Jinja2 expressions like {{ steps.fetch_data.output.body }}.

The execution model was simple: walk the sorted list, execute each step, persist the state to PostgreSQL, move to the next. No branching. No decisions. A conveyor belt that worked perfectly for linear workflows.

Then I needed conditions.

Why the graph stays flat

The workflow definition needed to stay simple: a flat list of steps with plain depends_on references. Conditions should be a step type like any other. The branching logic should live at runtime, not in the graph structure.

Most workflow engines encode branching directly into the graph. BPMN uses gateway nodes that split the flow into separate paths. Airflow uses BranchPythonOperator to select downstream tasks. You define the branches as part of the DAG, each path continues independently, and eventually they converge at a join node.

I sketched this out and hit two problems. A 12-step workflow with three conditions and two paths each produces a graph where users need to reason about up to eight distinct execution paths when designing the workflow. The depends_on array in each step definition becomes a tangled web of conditional references. Step 10 depends on step 7, but only if step 7's condition was true. The dependency model needs to become richer than "this step waits for that step."

The second problem is structural. The DAG parser in parser.py already handles cycle detection and depth validation on the flat dependency graph. Introducing branch paths would require a second validation pass for branch connectivity, dead-end detection, and convergence verification. Every new validation rule is a new failure mode for users to debug.

The skip set

The solution was to keep the topological sort and add a mutable skipped_steps set that grows during execution. When the orchestrator reaches a condition step, it evaluates the Jinja2 expression and calls _resolve_condition_branches(). That function reads the condition's true_branch and false_branch config (each a list of step IDs) and adds the non-taken branch's steps to the skip set.

async def _resolve_condition_branches(
    self,
    step_def: StepDefinition,
    output: dict,
) -> set[str]:
    skipped = set()
    result = output.get("result", False)
    if result:
        skipped.update(step_def.config.get("false_branch", []))
    else:
        skipped.update(step_def.config.get("true_branch", []))
    return skipped

Before executing each step in the sorted order, the orchestrator checks: is this step ID in skipped_steps? If yes, transition its state to skipped and move on. The skipped state is terminal in the state machine (empty transition set, no further changes allowed). The topological order never changes. The graph structure never changes. Steps just get removed from the plan as it runs.

A 12-step workflow with three conditions still has 12 entries in the topological sort. The orchestrator always walks the same list. It just skips some entries based on runtime evaluation. No path explosion. No conditional dependency syntax. No gateway nodes.

Where the truthiness got messy

The condition step evaluates a Jinja2 expression and gets back a string. Jinja2 doesn't return Python booleans from template rendering. Everything comes back as text. So the string "False" needs to be treated as falsy, and "1" as truthy.

I defined a _FALSY_VALUES frozenset in condition.py:

_FALSY_VALUES: frozenset[str] = frozenset({
    "", "false", "0", "none", "False", "None",
})

This handles the common cases: empty strings, Python-style false values, string representations of zero. But the orchestrator also needs to resolve which branch was taken, and it has its own truthiness check for the condition output. I ended up with the same logic in two places: the condition executor computes the boolean and writes it to the output, and the orchestrator reads that output to populate the skip set. Both places need to agree on what "false" means.

The duplication is a code smell I haven't fixed. The orchestrator should trust the result boolean from the condition's output (which the executor already computed) instead of re-evaluating truthiness on the raw expression result. It works today, but there's a subtle bug waiting if someone changes the falsy list in one place and not the other.

The state machine underneath

Making skip sets work required a state machine with six distinct step states: pending, queued, running, completed, failed, and skipped. The transitions are defined as an adjacency dictionary in state.py:

ALLOWED_STEP_TRANSITIONS: dict[str, set[str]] = {
    "pending": {"queued", "skipped"},
    "queued": {"running"},
    "running": {"completed", "failed"},
    "failed": {"queued"},
    "completed": set(),
    "skipped": set(),
}

Two details matter here. First, skipped has an empty transition set. Once a step is skipped, it stays skipped. You can't un-skip a step during execution. The condition evaluated, the branch was chosen, the decision is final.

Second, failed can transition back to queued. That's the retry mechanism. When a step fails and has retry attempts remaining (default 3, configurable up to 10 via RetryConfig), the orchestrator transitions it back to queued and re-executes. Each attempt updates step_exec.attempt in PostgreSQL before the retry runs. If the process crashes between retries, the attempt count survives in the database.

Every state transition writes to PostgreSQL before anything else happens. The orchestrator doesn't execute step N+1 until step N's final state is persisted. This is deliberately slow. An in-memory state machine would be faster, but if the worker dies between steps, every execution in flight would lose its state. Persist-before-execute means a crash leaves every execution in a known, recoverable position.

Cycle detection on the reverse graph

The skip set approach only works if the graph is actually a DAG. Cycles would cause the topological sort to loop forever. The detect_cycles() function in graph.py uses DFS with three-color marking: white (unvisited), gray (currently being explored), black (fully explored). A back edge from any node to a gray node means a cycle.

The non-obvious detail: the function builds a reverse graph where each entry maps a dependency to its dependents, not the other way around. If step C depends on step B, the reverse graph has B: [C]. DFS then explores "who depends on me?" paths. Cycles in that direction are what would break execution order. When a cycle is found, _reconstruct_cycle() walks backward through parent pointers to produce the full path: ["step_a", "step_b", "step_c", "step_a"]. The error message shows exactly which steps form the loop.

Maximum DAG depth is capped at 20 via a hardcoded MAX_DEPTH constant. The depth calculation uses Kahn's algorithm with a twist: instead of just sorting, it tracks the longest path to each node. If step D depends on both B and C, and B is at depth 2 while C is at depth 4, then D gets depth 5. The maximum across all nodes determines the workflow's depth.

What surprised me

I expected the hard part to be the topological sort or the cycle detection. Both were textbook implementations of well-known algorithms. The hard part was getting the skip set to interact correctly with depends_on.

Consider: step 9 depends on step 8, and step 8 is in the false branch of step 7's condition. If step 7 evaluates to true, step 8 gets skipped. But step 9 is still in the topological order. Its dependency was skipped, not completed. Should step 9 execute?

The answer I landed on: if all dependencies of a step are either completed or skipped, the step can execute. A skipped dependency counts as resolved for scheduling purposes. This lets you build workflows where step 9 runs regardless of which branch step 7 took, as long as its other dependencies are met. The orchestrator checks this when deciding which steps to queue next.

I could have made skipped dependencies block execution. But that would mean a single condition deep in the DAG could freeze all downstream steps, even ones that don't care about the branch result. The current behavior is more useful: skip sets remove steps from the plan, but they don't create invisible walls in the dependency graph.

The numbers

The engine handles workflows up to 50 steps with a maximum DAG depth of 20 in the longest dependency chain. Condition branching adds zero overhead to the execution plan because the topological sort runs once, before any step executes. The skip set is a Python set() with O(1) lookups. State persistence adds one database write per step transition, which at 15 connections in the pool (5 steady plus 10 overflow) handles the concurrency comfortably.

544 tests cover the engine, including edge cases for nested conditions (a condition inside another condition's true branch), skipped steps with downstream dependents, retry of a failed step after a prior execution was replayed, and sub-workflows that inherit the parent's depth counter. The condition branching logic alone accounts for 12 dedicated test cases covering truthiness evaluation, branch resolution, and the interaction between skip sets and the dependency resolver.

Why the PO Resolver Has Five Strategies, Not One Algorithm

Kingsley Onoh — Sat, 18 Apr 2026 11:12:22 +0000

PO-2026-001. PO 2026 001. po2026001. PO#2026-001. 2026-001.

Every one of those strings refers to the same purchase order. The first came from our internal PO system. The rest came from vendors. One vendor's accounting package strips dashes. Another inserts hash prefixes. A third just writes the sequence number and assumes the buyer will figure it out. A database equality query on po_number finds zero matches on four of those five.

That's the starting condition for the matching engine. The po_reference field on the invoice exists, is populated, and does not equal any PO number in the database. The naive response is to call the invoice unmatched and let a human deal with it. The whole point of the system is to not do that.

The Matcher's Real Job

The pipeline I inherited from my own PRD had a single phase called "PO resolution." Given an invoice, find its PO. The implementation I started writing was also single-step: normalize both sides of the string, compare, return a match or null. Clean code, one function, one SQL query.

It handled about 60% of real invoices.

The remaining 40% broke in three different ways. Some vendors omitted the PO reference entirely, writing it into the line item description instead. Some vendors used their own internal sales order number, which had no textual overlap with our PO number at all. Some vendors sent the right reference but with enough format noise that neither exact nor normalized comparison found it.

Three different failure modes. One matching algorithm. The ratio was not going to get better.

The Cascade, Not a Fallback

I rewrote PoResolver as an ordered chain of strategies, each one with a decreasing confidence ceiling. The signature looks the same from the outside: give me a tenantId, a poReference, a vendorId, an invoiceAmount, and an invoiceDate, and I'll hand back a PoResolution with a PO ID, a confidence score, and a method label. What changed is the inside.

// Strategy 1: Exact match
if (poReference != null) {
    val exact = tenantPos.firstOrNull { row ->
        row[PurchaseOrderTable.poNumber] == poReference
    }
    if (exact != null) {
        return@newSuspendedTransaction PoResolution(
            poId = exact[PurchaseOrderTable.id],
            confidence = 1.0,
            method = "exact",
        )
    }
}

// Strategy 2: Normalized match
if (poReference != null) {
    val normalizedRef = normalize(poReference)
    val normalized = tenantPos.firstOrNull { row ->
        normalize(row[PurchaseOrderTable.poNumber]) == normalizedRef
    }
    if (normalized != null) {
        return@newSuspendedTransaction PoResolution(
            poId = normalized[PurchaseOrderTable.id],
            confidence = 0.95,
            method = "normalized",
        )
    }
}

The full resolver in PoResolver.kt runs five of these in order. Exact at confidence 1.0. Normalized (strip whitespace, remove the PO prefix, lowercase) at 0.95. Jaro-Winkler fuzzy match against every PO in the tenant above 0.70, with the Jaro-Winkler score itself becoming the confidence. Vendor plus amount within 5% tolerance at 0.65. Vendor plus date within 90 days at 0.50. Then no match at 0.0.

Each strategy catches a failure the previous one could not.

Why the Confidence Decays on Purpose

The obvious objection to a cascade is that you could just run the weakest algorithm first and skip the rest. Fuzzy match solves both the exact case and the noisy case, right?

It doesn't. Not for the reason people expect.

Jaro-Winkler's 0.70 threshold is tuned for normal PO numbers. Two different POs issued a week apart to the same vendor will often score 0.78 against each other. PO-2026-001 and PO-2026-008 share the prefix, share the date segment, and differ only in the sequence number. Run fuzzy match first and you will occasionally match an invoice to a PO that is not the correct one. The system will have no way to know it's wrong, because the score says "probable match."

Running exact match first means any invoice that quotes the correct PO number unambiguously gets it unambiguously, at confidence 1.0. The fuzzy matcher never runs on the 60% of invoices where exact works. It only runs on the 20% where no exact or normalized match exists, and among those, only the cases with a close-enough string similarity get scored. The cascade constrains the search space before the weaker algorithms touch it.

Confidence 0.95 for normalized, 0.70-ish for fuzzy, 0.65 for vendor+amount, 0.50 for vendor+date. Those numbers are not arbitrary rankings. They are honest confidence statements about the evidence the match is built on. A 0.50 vendor+date match means: this invoice arrived from a vendor we have a PO with, issued within 90 days of the invoice date. That is weak evidence. It might be the right PO. It's enough to avoid the unmatched state, enough to surface for human review, and honest about how certain the system is.

The Normalization Function That Does Three Things

normalize() is short enough to fit on a screen. It strips whitespace, regex-removes the PO prefix variants, and lowercases. That's it:

private val PO_PREFIX_REGEX = Regex(
    """^(PO[-#\s]*)""",
    RegexOption.IGNORE_CASE,
)

internal fun normalize(value: String): String {
    return value
        .trim()
        .replace(PO_PREFIX_REGEX, "")
        .replace("\\s+".toRegex(), "")
        .lowercase()
}

Three lines of transformation. It cost me two afternoons.

What surprised me in the data was how many ways the same four characters can vary. PO-, PO, PO#, PO., po-, P.O.-, P.O.. The regex handles the first four. The last three we either accept as exact-match failures (they won't pass normalization either) or catch with fuzzy match at a lower confidence. I deliberately did not try to normalize every typographic variation. The regex would grow and the edge cases never end.

The better question was: what fraction of real PO references in the database fail normalization? I wrote a one-off script against a synthetic test set of 2,000 generated invoice references and measured how many failed each strategy in turn. Exact caught 58%. Normalized added another 27%. Fuzzy caught another 8%. Vendor+amount caught 3%. Vendor+date caught 2%. Unmatched: 2%.

The 58% exact match result was not a surprise. Most vendors copy-paste the PO number into their system once and then their system emits the same string forever. The 27% for normalized was the argument for writing the regex at all. The 3% vendor+amount rescue was the argument for keeping strategies 4 and 5 even though they look weak.

The One That Has No PO Reference at All

Strategies 4 and 5 run with no poReference string. The invoice arrived without a PO number written on it. Strategy 4 asks: are there any open POs for this vendor whose total matches the invoice total within 5%? Strategy 5 asks: are there any open POs for this vendor issued within 90 days of the invoice date?

The confidence on these is deliberately low. 0.65 and 0.50. Both are below the auto-approve threshold (0.95) and below the standard-approval threshold (0.70). A match at this confidence routes to escalated review, every time, with no exception. Which is the right behavior.

What these strategies buy you is the difference between "unmatched, no PO" and "possibly matched, here's a weak candidate." The human reviewer gets a starting point. Instead of opening the invoice in an empty state and searching manually, they open it with a suggested PO pre-attached and a 0.50 confidence label that tells them: we think this is the one, don't trust us.

I was wrong about this initially. My first instinct was that low-confidence matches are worse than no match at all. A human opening an invoice with a suggested PO will often accept the suggestion without checking. That's a real risk. I solved it at the approval routing layer: any match below 0.70 confidence is force-routed to the escalated queue, and the approval UI shows the confidence score prominently with the breakdown. The reviewer sees {po_match: 0.50, method: "vendor_date"} and knows to verify the PO number manually before clicking approve.

The system does not make the low-confidence decision. It surfaces the low-confidence candidate.

The One Tradeoff I Accepted

The cascade loads every PO for the tenant into memory on every invoice. For the normalized and fuzzy strategies, there is no index I can use: normalization happens after the row is in memory, and Jaro-Winkler needs the full string of both candidates. The query is SELECT * FROM purchase_orders WHERE tenant_id = ? and then the filtering is in Kotlin.

At 10,000 active POs per tenant, this is fine. The loop is a few milliseconds. At 100,000 active POs, it would start being noticeable. At 1 million, the cascade would need a different shape: pre-compute a normalized PO number column, index it, and do the normalized lookup in SQL. The fuzzy match would need a trigram index (pg_trgm is already enabled in migration V010) backing a SIMILARITY query.

I have not built that yet. The largest tenant on the system has about 3,000 open POs at any time, and the current approach takes under 10 milliseconds per invoice on that workload. Building the indexed version now would be optimization ahead of the constraint.

The scale plan is in the code comments. When a tenant crosses the threshold, the resolver gets a second implementation path. The interface stays the same.

What the Match Record Actually Stores

Every match result writes a JSONB breakdown. {po_match: 0.95, method: "normalized"}. The method string is the strategy that matched. That field is what makes the whole design defensible to operators.

When a finance lead looks at a low-confidence invoice in the approval queue, they don't just see "0.73 confidence." They see the breakdown:

{
  "po_match": 0.95,
  "line_match": 0.72,
  "receipt_match": 0.50,
  "price_match": 0.78,
  "overall": 0.73
}

Plus the PO resolution method label: "normalized". Now the reviewer knows the PO was found with reasonable confidence (0.95, normalized match), but the receipt verification dropped to 0.50 (no receipt found on that PO line yet), which pulled the overall score down. The remediation is obvious: wait for the goods receipt and rerun matching. Not reject the invoice.

That information existed in the old binary matcher too. It just wasn't exposed. The system knew why a match was weak and threw the reason away. Writing it to JSONB was the cheap part. Deciding to write it at all was the design decision.

What I'd Reconsider

The 0.50 confidence floor on vendor+date matching is doing work, but it's doing work that's not fully visible in the match record. A "vendor_date" method label with 0.50 confidence tells an operator that the invoice had no PO reference and we fell back to the weakest strategy. It does not tell them whether the vendor has three other open POs in the same date range that were almost as good a match.

If I redid this, I'd return the top three candidates from each strategy, not just the best one. The alternatives field on ConfidenceScore exists for exactly this and is currently always empty. The code comment in ConfidenceScorer.kt references it as planned work. An invoice that fell to strategy 4 with three equally plausible POs should surface all three to the reviewer with their individual confidence scores. Right now the reviewer sees one candidate and either accepts or searches manually.

That's the kind of detail that looks trivial and isn't. A single weak candidate is worse than a ranked list of weak candidates, because the single candidate suggests certainty the system doesn't have. Building the alternatives list is two afternoons of plumbing. Shipping without it is the tradeoff I made to hit the first release. It's the first thing I'd add in a second pass.