In most systems, a notification is a convenience. It tells you something happened. If it arrives late, or not at all, the user refreshes the page and moves on.
That is not the class of problem I was solving.
Fraud Sentinel is a fraud case management and RBI FMR reporting platform used by NBFCs and banks. When a fraud case is created, updated, or breaches its turnaround time, the right people must be notified, not as a courtesy, but as a process requirement embedded in compliance workflows. Whistle Sentinel is an enterprise whistleblowing portal. When an investigator sends a status update, or a reporter submits additional evidence through the case chat, the involved parties must receive that signal with the same reliability as a transaction confirmation. TrueScale is a legal metrology compliance tracker. Deadlines matter. Breaches matter. Silence is not an acceptable delivery state.
Three regulated products. Three distinct event vocabularies. One shared infrastructure underneath all of them. They share one fact: when something happens, somebody must be told, and the telling cannot be lost.
This is the story of how that infrastructure was designed, where the standard approaches broke down, and what it actually took to build notifications that cannot be lost.
The Event Surface Is Wider Than You Think
The first instinct when building notification systems is to enumerate the events. Case created. Case updated. That is usually where the list ends.
In practice, the event surface across these three products is significantly wider.
For fraud case management: a case can be created, updated, assigned to a new investigator, reassigned to a new department, have a form submission added to it, reach its TAT deadline, or breach it entirely. Each of these is a distinct moment that may or may not require a notification, depending on rules configured per tenant.
For the whistleblowing portal: an investigator sending a case status update is an event. A reporter submitting a new message in the case chat is an event. A reporter submitting additional supporting details is an event. These are not the same. The recipients differ, the message templates differ, the urgency differs.
For legal metrology compliance: a deadline approaching is different from a deadline breached. Both require different notification postures.
What this surface reveals is that hardcoding event types is a losing strategy from the start. The system needs to be configurable at runtime, where tenant administrators can define rules, conditions, and actions without a deployment. This led to a rules engine with three distinct rule types. Creation rules fire when a case or incident is first created. Update rules fire when a case or incident is updated, with trigger conditions scoped to specific update events such as form submissions, assignments, or status changes. Time-trigger rules fire based on TAT deadlines, calculated against configurable date fields or system-tracked timestamps, with configurable pre-offset and post-offset windows.
Each rule has conditions and actions. Conditions determine whether this particular case, at this particular moment, qualifies for the rule to fire. Actions determine what happens: which channels to notify, which recipients to resolve, which template to render. The engine is shared across all three products. Adding a new event type to any product is a matter of defining a new rule trigger, not modifying the pipeline.
The Rules Engine, and the Insight About Loading
Before the outbox, a word about what produces these notifications, because it shaped everything downstream.
Notifications are not hardcoded. They are emitted by a rules engine that clients configure at runtime. Each rule has conditions and actions. Conditions are predicates over static case fields, over submitted form-field values, and over entity hierarchy mappings: whether a case belongs to a department, a region, or a reporting line. Actions are notification actions: send an email, raise an in-app system notification.
This is the classic rules-engine shape, conditions and actions externalised from code. Martin Fowler's warning about rules engines is worth quoting here because I think it is correct: the central pitch, that "it will allow the business people to specify the rules themselves," he writes, "can sound plausible but rarely works out in practice," and the implicit chaining behaviour of a general engine "can easily end up being very hard to reason about and debug." His conclusion is to keep rules "within a narrow context" and prefer "a more domain specific approach." That is exactly what this is. There is no chaining. A rule's action never mutates state that another rule's condition reads, so the entire class of debugging nightmares Fowler describes simply does not exist here. The engine evaluates, collects matching rules, and emits notification intents. Nothing more.
The non-obvious engineering decision is in how conditions are evaluated. The obvious approach is to load the case, the whole aggregate with its form submissions and hierarchy mappings, and then run each rule's conditions against the in-memory object. This is what almost everyone does, and it is wasteful in a way that compounds. A case with dozens of form fields and a deep hierarchy is an expensive load, and most rules touch two or three fields.
The engine inverts this. It first inspects the conditions across the rules that could fire and determines which types of data are actually referenced: static fields, specific form fields, particular hierarchy levels. Only then does it issue queries, in parallel, for exactly those fields. If no rule references form submissions, the form-submission query never runs. This is targeted loading, the same principle that the rules-engine literature calls lazy fact loading: the recognition that performance "degrades with very large rule sets or expensive fact lookups" and that the cure is to not load facts you will not test. In a multi-tenant system where the same evaluation runs across many schemas under load, the difference between loading the whole case and loading three fields is the difference between a pipeline that keeps up and one that falls behind.
How the Design Actually Evolved
The right design is not the one I started with. This section is the honest account of how the notification system was built, then broken in a specific and instructive way, and then rebuilt properly.
The standard industry approach to this problem is straightforward: write the business entity to the database, commit the transaction, and then enqueue a job or call a messaging system to trigger downstream processing. Most teams do this and most of the time it works. The failure mode is silent and rare, which is the worst combination. The database commits, something goes wrong between the commit and the enqueue, and the event is gone. No error surfaces, because the HTTP response already returned success. The system has no record that the event happened, so there is nothing to retry and nothing to alert on. In most domains, a dropped notification is a minor inconvenience. In a regulated compliance system, it is an incident.
Stage one: EventEmitter only. The original implementation used NestJS's EventEmitter2. When a case was created, the creation service emitted a case.created event after committing the transaction. A listener picked it up and ran the rules engine, which triggered notifications directly. There was no outbox at this stage. The failure mode was silent and I encountered it firsthand: the database transaction committed, the notification never fired, and nothing in the system recorded that it had been missed. There was nothing to retry, nothing to alert on. The case existed. The notification did not. Nobody knew.
That is the moment I understood what the requirement actually was. Lost notifications are not an edge case to handle. They are the central problem to design around.
Stage two: EventEmitter with outbox bolted on. After learning about the transactional outbox pattern, I introduced a system_outbox_events table and had the rules engine write to it. But the EventEmitter architecture remained. The sequence was: commit the transaction, emit the event, listener picks it up, listener writes to the outbox. The problem was that the outbox write was happening inside the EventEmitter handler, which ran after commitTransaction() had already been called. The case was committed. The outbox write was now outside any transaction. If that write failed, for any reason, the case existed with no intent ever recorded. The atomicity I thought I had was an illusion. The outbox was present, but it was not transactional. Adding a table to the system is not the same as adopting the pattern.
Stage three: The refactor, and the proper pipeline. During the full codebase refactor, I removed EventEmitter from the notification path entirely. The pipeline was rebuilt in proper layers. Rule evaluation and the outbox write were moved inside the same database transaction as the case write. A BullMQ processor reads from the outbox, not from an event. A separate poller manages the outbox lifecycle across all tenant schemas. A dispatcher routes by product code. Preparation processors resolve recipients and placeholders before delivery. Each layer has one responsibility and hands off to the next. The architecture was not arrived at in one step. It was the result of understanding, through production failures, what "durable" actually means in a regulated system.
The rest of this article describes the final architecture.
Why an Outbox, and Which Outbox
The transactional outbox, as catalogued by Chris Richardson, is the pattern where you "store the message in the database as part of the transaction that updates the business entities. A separate process then sends the messages to the message broker." The outbox row is an intent, not a rendered message. It records that something happened and which rule fired, with enough context to produce a notification later. It does not yet know who will receive an email or what the email will say. That deferral is deliberate, and the reason for it comes later.
A separate process reads committed outbox rows and dispatches them. There are two well-known ways to build that reader, and the distinction is the single most consequential one in the whole design. Richardson catalogues them as the Polling Publisher, "publish messages by polling the database's outbox table," and Transaction Log Tailing, "tail the database transaction log and publish each message/event inserted into the outbox." Log tailing, implemented with a tool like Debezium reading PostgreSQL's write-ahead log, is the more scalable and lower-latency choice at very high throughput. It imposes no query load on the database and, because the WAL is in commit order, it delivers events in commit order.
I chose polling deliberately, and the multi-tenancy is the reason. Log-tailing tools read the WAL at the database level. Mapping a single physical WAL stream back onto per-tenant schemas, and operating Debezium and Kafka as additional infrastructure for a solo-maintained platform, is a large operational commitment. Polling is, in Richardson's words, the pattern that "works with any SQL database," and it kept the entire pipeline inside the stack I already ran: PostgreSQL, Redis, NestJS, with no new moving parts to operate. The honest cost is twofold. Polling has a latency floor set by the poll interval, and it imposes constant query load. For deadline notifications measured in days, a five-second floor is irrelevant. That judgment was made explicitly rather than discovered later, which is the difference between a tradeoff and a bug.
It is also worth being clear about what the outbox does not guarantee. Because the poller reads committed rows and claims them by status rather than reading from a commit-ordered write-ahead log, strict global ordering of notifications is not guaranteed. For this domain, that is acceptable. Notifications are individually meaningful events, not a stream that must be replayed in exact order.
The Multi-Tenant Wrinkles, Where the Real Lessons Are
A schema-per-tenant system routes queries by setting the PostgreSQL search path, the ordered list of schemas PostgreSQL searches to resolve an unqualified table name. The way you set it is not a detail. It is the single most consequential decision in the entire codebase, and getting it wrong leaks one client's data to another.
The trap is SET search_path versus SET LOCAL search_path. Plain SET search_path changes the setting for the session. With a connection pool, that means it persists on the physical connection after the request finishes and the connection returns to the pool. The next request to acquire that connection inherits the previous tenant's search path. In a multi-tenant fintech platform, that is cross-tenant data bleed: a query intended for tenant B silently reads tenant A's schema. The PgBouncer literature is unambiguous about this: "this connection is reused also for other services, so the services influence each other," and the fix is precisely SET LOCAL search_path. SET LOCAL scopes the change to the current transaction. When the transaction ends, PostgreSQL reverts it automatically, and the connection returns to the pool clean. Every tenant-scoped operation in this system runs inside a transaction with SET LOCAL search_path. There is no exception to that rule, because the cost of an exception is a confidentiality breach.
That correct decision then breaks something else, which is how I learned the next lesson. The outbox writer needs an idempotent insert: a dedupe key with "insert if not present, otherwise do nothing," so that a retried job cannot create a second copy of a notification. TypeORM exposes .orIgnore() for exactly this purpose, an ON CONFLICT DO NOTHING wrapper. But with the search path set via SET LOCAL, TypeORM's query builder did not reliably target the tenant schema for the conflict-handling insert. The fix was raw SQL: an explicit INSERT ... ON CONFLICT (dedupe_key) DO NOTHING, run on the same transaction-scoped connection that holds the SET LOCAL. The dedupe key encodes the case, the action, the channel, and the event type, making it deterministic and collision-resistant across retries.
The dedupe key is what makes the whole at-least-once pipeline safe. A polling relay provides at-least-once delivery, not exactly-once. Richardson states it plainly: the relay "might publish a message more than once. It might, for example, crash after publishing a message but before recording the fact that it has done so. When it restarts, it will then publish the message again. As a result, a message consumer must be idempotent." This is the Idempotent Consumer pattern, and the dedupe key is its concrete implementation. You do not fight duplicate delivery. You design so that a duplicate changes nothing.
There is a smaller, sharper trap, from Redis. The poller and dispatcher coordinate through Redis, and I run ioredis with a keyPrefix so that keys are namespaced. The footgun: ioredis's keyPrefix is applied to ordinary key commands but not to KEYS, SCAN, or EVAL. This is a long-standing documented behaviour, with open issues going back years. If you rely on the prefix being universal, your SCAN silently matches nothing and your Lua locking script operates on the wrong key. I prepend the prefix manually for those commands. It is not elegant. It is correct, which matters more.
The Pipeline
With the outbox as the foundation, the full pipeline has five distinct layers, each with a well-scoped responsibility.
The rules processor. When a case is created or updated, a BullMQ job is enqueued for the rules processor. The processor evaluates active rules against the case and, for each matched rule with notification actions, writes an intent row into system_outbox_events. Both the rule evaluation and the outbox write happen within the same database transaction. That is the entire point.
The rules processor does not load the entire case data. A case can have dozens of form field submissions, entity hierarchy mappings, and static field values. Loading all of them to evaluate a rule that references two fields is wasteful in a way that compounds under load. Instead, the processor inspects the conditions across all candidate rules first, determines which fields are actually referenced, and fires targeted parallel queries only for those fields. Static case fields, form field submissions, and entity hierarchy mappings are three separate queries, each scoped to the IDs present in the conditions. If no rule references entity mappings, that query does not run.
For update rules, the processor applies a deliberate four-phase sequence before loading any case data. First, it filters by trigger type at the database query level, so a rule configured for form submissions is never evaluated when the trigger was an assignment change. Second, it filters by metadata in memory, eliminating rules that reference a specific form when a different form triggered the update. Third, it evaluates trigger conditions against current case state using targeted loading. Fourth, it evaluates filter criteria against the survivors. Each phase eliminates candidates before the more expensive next phase runs. Trigger type filtering costs a query predicate. Metadata filtering costs nothing. Case data loading only happens for rules that survived both.
For time-trigger rules, the evaluation is different. A TAT cron runs on a schedule. It loads active time-trigger rules, evaluates each rule's TAT type against its configured offset window, finds the matching cases, and for each writes notification intents to the outbox. Each rule gets its own QueryRunner. A failure in one rule's evaluation should not roll back the work done by other rules. Isolation at the rule level is the right boundary.
The outbox poller. The poller wakes every five seconds. Five seconds is a deliberate compromise: it is the latency floor, traded against database query load, and for deadline notifications measured in days it is fast enough while keeping the query rate modest.
The poller has four properties that took the most design thought, and each exists because of a specific failure mode I either hit or refused to risk.
A distributed Redis lock ensures that when the service runs multiple instances, only one poller processes a given schema at a time. This is a single-instance Redis lock, not the multi-node Redlock algorithm, and that is the appropriate choice here. The lock is a performance optimisation to prevent two workers from contending on the same rows, not the sole guarantee of correctness. The correctness guarantee lives in the database, in row-level claiming and the dedupe key. A lost lock at worst causes a duplicate attempt, which the dedupe key absorbs. The lock makes the common case efficient. The database makes every case correct.
A circuit breaker ensures that when something is systematically wrong, the database is unreachable or an entire schema's processing is failing, the poller stops hammering it. When failures exceed a threshold, the breaker trips to open and rejects calls immediately rather than "waiting for the operation to time out or never return," then probes for recovery. Without it, a poller waking every five seconds against a failing dependency becomes a load amplifier exactly when the system can least afford it.
Round-robin rotation through tenant schemas is the property that is invisible until it bites. The poller does not process tenants in a fixed order. It keeps a cursor and rotates, so each cycle starts where the last left off. With a fixed order and a bounded amount of work per cycle, under load the poller spends its entire budget on the first N schemas every single time. The schemas at the tail of the list are never reached. They do not fail loudly. They simply never receive their notifications, which in this domain means a tenant silently stops receiving regulatory deadline warnings while every other tenant is fine. Round-robin guarantees that every schema is reached in bounded time regardless of its position in the list.
Lease reclaim for crashed workers addresses the last failure mode. When the poller claims an outbox row it marks it PROCESSING and stamps a lease timestamp. When the dispatcher forwards it, the row moves to DISPATCHED with its own lease. Both leases exist to answer one question: what happens when a worker crashes mid-job? Without leases, a row marked PROCESSING by a worker that then dies is stuck in that state permanently. It is claimed but will never complete. The notification is lost as surely as in the naive design, just by a different route. The lease makes the claim expire. A later poll cycle sees a PROCESSING row whose lease has elapsed, concludes the worker is gone, and reclaims it. The lease duration is the maximum time a notification can be stuck behind a dead worker, and you tune it accordingly.
The dispatcher. The dispatcher is a BullMQ processor that receives a claimed outbox event and routes it to the correct preparation queue. The routing key is product_code. Fraud Sentinel cases route to case preparation queues. Whistle Sentinel incidents route to incident preparation queues. The same dispatcher handles both, and adding a third or fourth product is a matter of adding a routing case.
This layer is defined by what it refuses to do. The products do not know about each other at the queue level. There is no shared queue that all three drain. Instead there is one shared mechanism, the outbox, the poller, the dispatcher, and cleanly separated channels per product. A backlog or a poison message in one product's preparation queue cannot stall another product's notifications. This is the seam that lets the platform add a product without touching the others.
The preparation processors. Each channel has its own processor per product. A preparation processor picks up a DISPATCHED outbox event and does the work the outbox intent deliberately left undone. It resolves recipients, which may be the case's assigned users, the heads of a department, specific user IDs, or custom email addresses configured on the rule. It resolves the notification template, substituting placeholders with real values drawn from static case fields, entity hierarchy values, and form field submissions. Then, and only then, it enqueues the actual delivery job.
The most important decision in the whole design lives here: recipient resolution happens in the preparation processor, not in the rules engine at the moment the rule fires. It would be simpler to resolve recipients when the rule matches and store the list in the outbox row. That would be wrong. Recipients are not stable between the moment a rule fires and the moment a notification is delivered. A department head changes. A user is deactivated. An assignment is reassigned. If you bind recipients at rule-evaluation time and the notification is delivered minutes or hours later, entirely possible with TAT rules, retries, and circuit-breaker pauses, you deliver to a stale audience: emailing someone who has left, or failing to email the person now responsible. Resolving recipients at preparation time, as late as possible before delivery, binds to the current truth. This is late binding applied to people, and in an organisation where responsibility moves between humans it is the difference between a notification that reaches the right person and one that reaches a ghost.
There is one failure-handling pattern here that is easy to get wrong. A preparation processor does its work inside a transaction. If that transaction fails and rolls back, you naturally want to mark the outbox event as FAILED so it is visible and retriable. But you cannot do that on the rolled-back transaction's query runner. It is poisoned. Any further statement on it is part of the aborted transaction and will not persist. The fix is to mark the failure on a fresh query runner, in its own separate transaction, deliberately outside the failed one. It feels redundant the first time you write it. It is the only way the failure record actually commits.
The delivery queues. The final delivery queues are deliberately dumb. They receive a fully resolved subject, body, and recipient list. Email delivery sends the email. In-app notification delivery writes the notification record. Neither knows anything about cases, rules, or tenants beyond the schema context passed in the job payload. That simplicity is the point. Complexity belongs upstream, where it can be reasoned about. By the time a job reaches this layer, every decision has already been made.
The Shared-Infrastructure Dividend
Step back and the architecture is one pipeline: rule evaluation writes intents to a shared outbox inside the business transaction; one poller, with its lock, breaker, rotation, and leases, drains the outbox safely across all tenants; one dispatcher routes by product code; per-product preparation processors resolve and deliver. The same machinery serves all three products.
The dividend is concrete. When a fourth product is added to the platform, it does not build a notification system. It defines its rules, its templates, and its preparation processors for the channels it needs, and it inherits for free: durable intent capture, crash-safe delivery, tenant isolation, fairness across schemas, idempotency, and failure visibility. The hardest parts of "tell someone, and never lose the message" are solved once, at the platform layer, and amortised across every product that will ever live on it. That is the entire argument for building it this way rather than as a feature inside one product. The second and third products paid almost nothing for notifications, and the fourth will pay less.
What I Would Tell Someone Building This
Take the dropped-message failure seriously first, before you design anything, because it is the requirement that forces the outbox and everything the outbox implies. If you can tolerate lost notifications, you do not need most of this and you should not build it. If you cannot, and in a regulated domain you cannot, then the outbox is not optional and the dual-write shortcut is a latent incident.
Put correctness in the database and use Redis and queues for speed, never the reverse. The lock, the leases, the circuit breaker, the round-robin: these make the system fast and fair and resilient, but every one of them can fail without losing a notification, because the durable intent and the dedupe key live in PostgreSQL under transactional guarantees. When you find yourself depending on Redis for correctness, you have made a mistake you will pay for during a network partition.
And resolve as late as you can. The outbox stores intent, not a rendered message, and recipients are bound at delivery time, because the world changes between cause and effect. Most of the subtle correctness in this system comes from deferring decisions until the last moment when the information is freshest, which is, in the end, the same discipline as the outbox itself: do not commit to an irreversible action until you are certain it will actually happen.
Caveats
This article describes one system as built, with its specific constraints: a solo maintainer, a stack of PostgreSQL, Redis, and NestJS, a regulated domain where lost messages are incidents, and a tenant count served comfortably by schema-per-tenant isolation. Several decisions are right for those constraints and would be wrong for others.
At very high throughput, log tailing beats polling and the five-second floor becomes intolerable. Past a few thousand tenants, schema-per-tenant strains PostgreSQL's system catalogs and a different isolation model is warranted. The single-instance Redis lock is sufficient only because correctness does not rest on it. A system that leaned on the lock for correctness would need Redlock and, even then, would inherit the unresolved debate about Redlock's safety under clock skew.
The pipeline described here handles three products, an arbitrary number of tenant schemas, and a configurable event surface that operators can extend without code changes. It has been running in production across fraud case management, enterprise whistleblowing, and legal metrology compliance workflows.
None of the individual techniques here are novel. The outbox pattern is well-documented. Distributed locking with Redis is standard. Schema-per-tenant is a known multi-tenancy strategy. The contribution, if there is one, is in the specific combination of constraints: regulatory stakes, schema isolation, shared infrastructure across distinct products, and the precise places where the standard approaches break down under those constraints. Those are the places worth examining carefully.
None of these are universal prescriptions. They are the record of a specific set of tradeoffs, made deliberately, under constraints I could name. That is the only kind of engineering writing I trust.
Top comments (0)