DEV Community: Tope Taiwo

Transactions, Events and the Problem of Reliable Side Effects

Tope Taiwo — Thu, 22 Oct 2026 08:32:21 +0000

"Understand the dual-write problem and how transactional outbox patterns, idempotent consumers, retries, and recovery improve event reliability."

A common backend operation looks simple:

Update database
      +
Publish event

The difficulty appears when one succeeds and the other fails.

This is a problem I consider when designing backend systems that coordinate database transactions with asynchronous processing, external services, or other independently operating components.

The challenge is not simply how to save a record or publish a message.

It is how to ensure that the system remains consistent when failures occur between those operations.

The dual-write problem

Suppose the application does:

1. Save order
2. Publish OrderCreated

If step 1 succeeds and step 2 fails, the database contains the order but downstream systems never receive the event.

The order may exist, but the systems responsible for inventory, notifications, reporting, or other follow-up operations may never learn that it was created.

Reverse the order and another failure mode appears.

1. Publish event
2. Save order

If the database operation fails after the event has been published, consumers may process an event for a transaction that never committed.

The problem exists because the application is trying to coordinate two independent operations without a shared atomic commit.

This is commonly known as the dual-write problem.

The two operations may each be reliable on their own. That does not make the combination reliable.

Transactions do not automatically solve messaging

A database transaction can provide atomicity within the database.

For example:

BEGIN
  │
  ├── Insert Order
  ├── Insert Order Items
  └── Update Inventory
  │
COMMIT

If an operation fails before the transaction commits, the database can roll back the changes covered by that transaction.

This is valuable because related business changes remain consistent.

However, a database transaction cannot automatically make an external message broker part of that same transaction.

Consider:

await prisma.$transaction(async (tx) => {
  const order = await tx.order.create({
    data: orderData,
  });

  await messageBus.publish({
    type: "OrderCreated",
    payload: {
      orderId: order.id,
    },
  });
});

At first glance, this appears to combine the database change and event publication into one operation.

It does not.

The message broker does not automatically participate in the PostgreSQL transaction.

If the message is published successfully but the database transaction later rolls back, consumers may process an event for an order that does not exist.

If the database commits but the application encounters a failure while publishing, the order may exist without the corresponding event.

Holding a database transaction open while calling an external service also increases transaction duration and can introduce unnecessary latency, resource contention, and failure dependencies.

The important distinction is that database atomicity does not imply atomicity across independent systems.

The transactional outbox pattern

This is where patterns such as the transactional outbox become useful.

The idea is straightforward: instead of attempting to publish an event directly as part of the database transaction, persist the intent to publish it in the same transaction as the business change.

Conceptually:

Database Transaction
      │
      ├── Business Data
      │
      └── Outbox Event
              │
              ▼
          COMMIT

Both are committed together.

A separate process can then publish the outbox event:

Outbox
  │
  ▼
Publisher
  │
  ▼
Message Bus
  │
  ▼
Consumers

If the transaction rolls back, neither the business change nor its outbox event is committed.

If the transaction commits, the outbox record remains available for publication, even if the messaging infrastructure is temporarily unavailable.

The publisher can retry delivery without requiring the original business operation to run again.

This changes the consistency problem.

Instead of trying to make a database write and a broker publication happen atomically, the application makes the business change and the intent to publish atomic. A separate process is responsible for delivering that intent.

The message may be published later, but the system has a durable record that publication is required.

A practical implementation

In a PostgreSQL-backed TypeScript application, the business record and outbox event can be persisted through the same Prisma transaction.

For example:

await prisma.$transaction(async (tx) => {
  const order = await tx.order.create({
    data: orderData,
  });

  await tx.outboxEvent.create({
    data: {
      eventId: crypto.randomUUID(),
      eventType: "OrderCreated",
      aggregateId: order.id,
      payload: {
        orderId: order.id,
      },
      status: "PENDING",
    },
  });
});

The example assumes an outbox model containing the corresponding fields.

The significant detail is that both operations use the same transaction client, tx.

If either operation fails, the transaction does not commit either change.

The outbox record might contain fields such as:

eventId: A stable identifier for the event.
eventType: The type of event to publish.
aggregateId: The identifier of the business entity associated with the event.
payload: The serialized event data.
status: The publication state.
createdAt: When the event was recorded.
publishedAt: When successful publication was recorded, if applicable.
attempts: The number of publication attempts.
lastError: Diagnostic information about the most recent failure.

The exact schema depends on the application's requirements.

The outbox is not simply a logging table. It represents durable work that remains to be delivered.

Its lifecycle, retention policy, indexing strategy, and recovery behavior therefore deserve the same attention as other operational components.

The publisher introduces another failure boundary

The transactional outbox solves the original dual-write problem, but it does not eliminate every failure mode.

Consider the publisher:

1. Read pending outbox event
2. Publish event
3. Mark event as published

What happens if the message broker accepts the event but the publisher crashes before step 3?

The outbox record still appears pending.

When the publisher retries, it may publish the same event again.

This means a transactional outbox generally supports reliable, retryable publication with possible duplicate delivery, not automatic exactly-once processing across the database and broker.

That distinction is important.

A robust implementation should assume that an event may be delivered more than once and design its consumers accordingly.

For example, a consumer can use the stable event identifier to detect duplicate processing. For database-backed business effects, a unique constraint on the processed-event identifier can help prevent the same logical event from being applied twice.

However, deduplication and external side effects require careful handling. Recording an event as processed and performing an external action are not automatically atomic either.

The outbox moves the reliability boundary. It does not remove the need to reason about it.

Consumer idempotency still matters

Suppose an OrderCreated event triggers an inventory reservation.

A consumer might receive the same event twice because the publisher retried it or the broker redelivered it.

Without appropriate safeguards, the inventory reservation could be applied twice.

A simplified processing design looks like this:

Receive Event
     │
     ▼
Check Event Identity
     │
     ├── Already Processed
     │       │
     │       ▼
     │   Skip Duplicate
     │
     └── Not Processed
             │
             ▼
       Apply Business Change
             │
             ▼
       Record Event Identity
             │
             ▼
          COMMIT

When the business change and processed-event record belong to the same database, they can often be committed in one transaction.

A uniqueness constraint on the event identifier provides an additional safeguard against concurrent duplicate deliveries.

If processing requires an external API call, however, the consumer must consider the possibility of a crash between the external operation and the local completion record.

Depending on the operation, a provider-supported idempotency key, a durable workflow, reconciliation, or another coordination mechanism may be necessary.

The correct approach depends on the business consequence of repeating the operation.

For example, generating a report twice may be tolerable. Applying the same financial adjustment twice may not be.

Idempotency is therefore not just a messaging concern. It is a property that should be considered when designing the business operation itself.

Polling versus change data capture

There is more than one way to publish outbox events.

Polling

A background publisher periodically queries the outbox table for pending events.

Outbox Table
     │
     ▼
Poll Pending Events
     │
     ▼
Publish to Broker
     │
     ▼
Update Publication State

Polling is relatively straightforward to implement and operate.

However, the implementation must account for concurrent publishers, batch sizes, event ordering, retry behavior, and the possibility of multiple workers attempting to claim the same record.

Database locking, appropriately designed claim operations, or other coordination mechanisms can help prevent unnecessary contention.

The polling interval also affects publication latency.

Change data capture

Another approach is to use change data capture (CDC) to observe committed database changes and feed outbox records into a messaging pipeline.

This can reduce the need for an application-managed polling loop and may be appropriate for systems with higher throughput or existing streaming infrastructure.

However, CDC introduces its own operational requirements, including connector management, replication-log retention, schema evolution, monitoring, and recovery.

Neither approach is universally better.

Polling can be a good starting point for a system with moderate throughput and straightforward operational requirements. CDC may be more suitable when the scale, latency requirements, or existing infrastructure justify the additional complexity.

The choice should follow the workload rather than the popularity of the pattern.

Event ordering and concurrent publishers

Reliable publication does not automatically guarantee that every consumer will observe events in the order the business produced them.

Suppose an order produces these events:

OrderCreated
OrderConfirmed
OrderDispatched

If multiple publishers process pending records concurrently, or the broker distributes messages across partitions or sessions, downstream processing may observe a different order unless ordering is explicitly supported and configured.

For many workflows, this is manageable because consumers validate the current business state or process events independently.

For workflows that require ordering, the design may need to preserve order per aggregate, use a stable partition or session key where supported, or prevent later events from overtaking earlier events for the same entity.

The requirement should be explicit.

Global ordering can constrain throughput unnecessarily. Per-aggregate ordering is often sufficient when events concern independent business entities.

The outbox schema and publisher strategy should reflect that distinction.

Retries, dead-lettering, and operational recovery

A publisher needs a clear response to failures.

Temporary broker unavailability may justify retries with exponential backoff. Invalid message contracts or permanently rejected messages may require investigation instead of endless retries.

A production design should establish:

Which failures are retryable.
How retry delays are calculated.
Whether retries have a maximum or continue until resolved.
How permanently failing events are isolated.
How operators inspect and replay failed events.
How long published records are retained.
How publication lag is monitored.

A dead-letter mechanism can isolate messages that cannot be processed automatically, but moving a message to a dead-letter queue does not resolve the underlying business problem.

Likewise, deleting a failed outbox record may remove the evidence that a required business event was never delivered.

Recovery should be deliberate and auditable.

Useful operational metrics include the number of pending events, the age of the oldest pending event, publication failures, retry counts, and the time between event creation and successful publication.

These signals reveal whether the asynchronous pipeline is keeping up with the business workload.

This connects to previous decisions

This is why database architecture and messaging architecture cannot always be designed independently.

The moment a business operation produces an external side effect, consistency becomes an architectural concern.

It also connects to how I approach application-level messaging boundaries.

An application should not need to know whether its event is eventually published through BullMQ, Azure Service Bus, or another messaging infrastructure. However, the application still needs a reliable way to record the obligation to publish that event.

The outbox provides that durable boundary between the database transaction and the messaging infrastructure.

The publisher can then implement the transport-specific behavior, while consumers handle idempotency and the business effects associated with the event.

These responsibilities are separate, but they must work together.

Not every operation needs an outbox. A best-effort cache refresh may not justify one. A business workflow that must not silently lose an event may.

The decision should reflect the consequences of failure.

Final thought

Distributed systems often fail at the boundaries between systems.

Understanding those boundaries is more valuable than memorizing patterns.

The outbox pattern is useful because it addresses one specific consistency problem: reliably connecting database state changes with asynchronous messaging.

It does not eliminate duplicate delivery, guarantee exactly-once execution, or make every external side effect atomic.

It gives the application a durable record of what must be published and a mechanism for recovering when the messaging infrastructure is unavailable.

That is the distinction that matters when designing reliable backend systems.

The goal is not to add patterns everywhere.

It is to understand which failure a pattern addresses, which guarantees it provides, and which responsibilities still belong elsewhere in the architecture.

Designing Background Jobs Without Infrastructure Coupling

Tope Taiwo — Wed, 21 Oct 2026 09:34:21 +0000

"Use messaging boundaries to keep application workflows independent of queue providers while preserving delivery guarantees, idempotency, and job semantics."

Background processing is an important part of backend systems.

Email delivery, notifications, reports, imports and other long-running operations do not always belong inside the HTTP request.

Some operations are too expensive to execute synchronously. Others depend on external services that may be slow or temporarily unavailable. In many cases, the API should accept a request, record the necessary business state, and allow a worker to complete the operation independently.

The architectural question is how to introduce asynchronous processing without coupling the entire application to one queue technology.

The direct approach

A service might do this:

await queue.add("send-email", payload);

That works.

For a small application, it may even be sufficient.

However, as the application grows, the queue client can start appearing throughout application services, controllers, and business workflows.

The problem is not the method call itself. The problem is the dependency it introduces.

If application services depend directly on BullMQ, for example, they become aware of concepts and behaviors associated with that infrastructure. Switching to Azure Service Bus may then require changes beyond the infrastructure layer.

The application may also become coupled to provider-specific options such as job priorities, delay settings, retry configuration, and scheduling APIs.

That is a dependency-management problem, not merely a queue-management problem.

Introduce a messaging boundary

A better boundary can be:

interface MessageBus {
  publish<T>(message: DomainMessage<T>): Promise<void>;
}

The application can then say:

await messageBus.publish({
  type: "EmailRequested",
  payload,
});

The infrastructure decides how to deliver it.

This follows the dependency inversion principle: application-level code depends on an abstraction, while concrete messaging infrastructure implements that abstraction.

The business operation expresses what needs to happen. The infrastructure determines how the message is transported.

A more complete TypeScript contract might look like this:

export interface MessageEnvelope<TPayload> {
  messageId: string;
  type: string;
  version: number;
  occurredAt: string;
  correlationId?: string;
  payload: TPayload;
}

export interface MessageBus {
  publish<TPayload>(message: MessageEnvelope<TPayload>): Promise<void>;
}

The envelope provides a stable place for message identity, schema versioning, timestamps, and request correlation.

These fields become useful when investigating failures across HTTP requests, background workers, and external services.

The version field also provides a foundation for evolving message contracts without assuming that every producer and consumer will be deployed simultaneously.

One important distinction: a message such as EmailRequested is generally an application or integration message, not necessarily a domain event. A domain event describes something meaningful that has happened in the business, such as OrderConfirmed. A command or request asks for an operation to be performed.

These concepts can share transport infrastructure, but they should retain their distinct meanings.

Separate publishing from processing

A messaging boundary is only one part of the architecture.

The consumer also needs a boundary between message delivery and business processing.

A simplified design looks like this:

HTTP Request
     │
     ▼
Application Service
     │
     ▼
MessageBus Interface
     │
     ▼
Messaging Adapter
     │
     ├── BullMQ / Redis
     │
     └── Azure Service Bus
                  │
                  ▼
              Consumer
                  │
                  ▼
          Application Handler
                  │
                  ▼
           Business Operation

The producer and consumer can now evolve independently of the underlying transport.

The HTTP application does not need to know how a worker receives messages. Likewise, the business handler does not need to know whether the message arrived through a Redis-backed queue or a cloud messaging service.

For example, a report-generation handler should be concerned with validating the request, generating the report, storing the output, and recording the result.

It should not need to know how BullMQ acknowledges a job or how Azure Service Bus settles a message.

Those concerns belong in the infrastructure adapter.

Why I care about this

I have specifically been designing systems where the messaging infrastructure should not be permanently tied to one provider.

For example:

MessageBus
   │
   ├── BullMQ / Redis
   │
   └── Azure Service Bus

The business operation should not have to change simply because the transport changes.

This is particularly useful when an application begins with a lightweight Redis-backed queue but later requires managed messaging infrastructure, different operational capabilities, or integration with other enterprise systems.

However, provider independence should be an architectural objective rather than a promise that switching providers will always be effortless.

The application should isolate provider-specific dependencies while explicitly defining the capabilities it requires.

A transport change may still require infrastructure configuration, deployment changes, operational adjustments, and changes to message routing.

The purpose of the boundary is to prevent those changes from unnecessarily spreading through the business logic.

Define the abstraction around application needs

An abstraction should describe the capabilities the application actually needs.

For example, if the application only needs to publish background work, a small interface may be sufficient:

export interface BackgroundJobPublisher {
  publish<TPayload>(message: MessageEnvelope<TPayload>): Promise<void>;
}

If the application needs delayed delivery, scheduling, or explicit routing, those capabilities should be introduced deliberately.

They should not be added merely because one provider exposes them.

Consider these examples:

// Application-level intent
await jobPublisher.publish(message);

Compare that with:

// Provider-specific behavior leaking into application code
await bullQueue.add("generate-report", payload, {
  attempts: 5,
  backoff: {
    type: "exponential",
    delay: 2_000,
  },
  removeOnComplete: true,
});

The second example is not inherently bad. Those options are useful when configuring BullMQ.

The architectural problem appears when business services must understand and reproduce those settings across multiple providers.

Instead, infrastructure configuration should translate application-level policies into provider-specific options.

For example, an application may define a retry policy for transient report-generation failures. The BullMQ adapter can translate that policy into its retry configuration, while an Azure Service Bus adapter can implement the appropriate delivery, redelivery, and dead-letter behavior supported by that platform.

The abstraction should remain small enough to be understandable and explicit enough to preserve the guarantees the application needs.

But abstraction must preserve semantics

A queue abstraction cannot pretend that every messaging platform behaves identically.

The design must account for the semantics the application actually needs.

This includes questions around:

Retries.
Ordering.
Acknowledgement and settlement.
Dead-lettering.
Idempotency.
Delivery guarantees.
Delayed delivery and scheduling.
Message expiration.
Concurrency and backpressure.

These differences matter because two providers may expose similar APIs while providing different operational guarantees.

For example, BullMQ manages jobs through Redis and provides job-oriented capabilities such as retries, delays, and concurrency controls.

Azure Service Bus provides managed messaging capabilities, including message settlement, delivery-count handling, and dead-letter queues.

A successful method call to publish a message also does not mean the business operation has completed. It means the message was accepted according to the publishing mechanism's guarantees.

The consumer may execute later, fail, retry, or encounter a permanent error.

The application must define what success means at each stage.

Delivery guarantees and idempotency

Distributed messaging systems can deliver the same logical work more than once.

A worker may complete an external operation and then fail before acknowledging the message. The infrastructure may redeliver the message because it cannot establish that processing completed successfully.

This means an application should not assume that a job handler executes exactly once merely because its queue has retries or acknowledgement support.

Consider a notification workflow:

Message Received
       │
       ▼
Send Notification
       │
       ▼
Record Completion
       │
       ▼
Acknowledge Message

If the process crashes after sending the notification but before recording completion, a retry could send the notification again.

An idempotent consumer is designed to handle repeated delivery without repeating the business effect unnecessarily.

One approach is to assign a stable message identifier and record processed messages in a durable store with a suitable uniqueness constraint.

However, a deduplication record alone does not make every external side effect atomic. If sending an email and recording its completion are separate operations, a crash can still occur between them.

For external effects, the design may need provider-supported idempotency keys, durable delivery records, reconciliation, or a workflow that explicitly handles uncertain outcomes.

The correct solution depends on the business operation.

For example, duplicate report-generation work may be acceptable if the output is safely replaceable. Duplicate financial transfers are not.

The messaging abstraction must not hide these differences.

The database transaction problem

One of the most important failure cases appears when an application writes to a database and publishes a message as part of the same business operation.

Consider this sequence:

1. Save Order to PostgreSQL
2. Publish OrderConfirmed

What happens if the database transaction commits successfully but the message publication fails?

The order exists, but downstream consumers may never receive the event.

Reversing the order does not eliminate the problem. The message might be published successfully while the database transaction subsequently rolls back.

This is the dual-write problem.

For business operations that require reliable event publication, the transactional outbox pattern is a common solution.

Database Transaction
        │
        ├── Save Business Record
        │
        └── Save Outbox Message
                  │
                  ▼
             Transaction Commit
                  │
                  ▼
            Outbox Publisher
                  │
                  ▼
              Message Bus
                  │
                  ▼
               Consumer

The business record and its outbox message are committed in the same database transaction.

A separate publisher reads pending outbox records and publishes them to the configured messaging infrastructure. After successful publication, it records the appropriate delivery state.

The publisher may still crash after sending a message but before marking the outbox record as published. Therefore, duplicate publication remains possible, and consumers should be designed accordingly.

The outbox pattern does not create a distributed transaction between PostgreSQL and the queue. It makes the intent to publish durable and recoverable.

It is particularly useful for order processing, approvals, financial workflows, and other operations where losing an event would create an inconsistent business state.

For a best-effort cache refresh or noncritical notification, the added complexity may not be justified. The reliability requirement should determine the pattern.

Publishing is not the same as processing

A clean messaging boundary should distinguish the acceptance of a message from the completion of its work.

For example:

await messageBus.publish(message);

This should not imply that the email has been delivered or the report has been generated.

It should mean that the publishing operation completed according to the contract defined by the adapter.

The application may return a response such as:

{
  "status": "accepted",
  "messageId": "msg_123"
}

The exact response depends on the API contract.

If the client needs to know when processing finishes, the application needs an additional mechanism, such as a persisted job-status resource, a notification, or a polling endpoint.

This distinction prevents an asynchronous API from promising synchronous completion.

It also helps separate three different concerns:

Acceptance: Was the work successfully submitted?
Execution: Did a worker process it successfully?
Business completion: Was the intended business outcome achieved and recorded?

Those states may occur at different times and should not be conflated.

Avoid building a generic messaging framework

There is a temptation to create an abstraction that supports every conceivable queue feature.

For example:

interface UniversalQueue {
  publish(): Promise<void>;
  schedule(): Promise<void>;
  cancel(): Promise<void>;
  retry(): Promise<void>;
  acknowledge(): Promise<void>;
  deadLetter(): Promise<void>;
  changePriority(): Promise<void>;
}

This can become a lowest-common-denominator abstraction that either exposes provider-specific details everywhere or fails to express the capabilities of individual providers accurately.

Not every messaging technology supports the same operations in the same way.

A better approach is to define a small core contract and introduce additional interfaces only when the application genuinely requires those capabilities.

For example:

interface MessagePublisher {
  publish<TPayload>(message: MessageEnvelope<TPayload>): Promise<void>;
}

interface ScheduledMessagePublisher extends MessagePublisher {
  publishAt<TPayload>(
    message: MessageEnvelope<TPayload>,
    scheduledAt: Date,
  ): Promise<void>;
}

Even this extension should be introduced only when scheduling is a real application requirement.

If scheduling semantics differ between providers, the contract must define what scheduledAt means, what happens when a message is late, and how cancellation or rescheduling works.

A smaller abstraction with explicit limitations is often more maintainable than a universal interface that promises capabilities it cannot consistently deliver.

Organize the TypeScript implementation around boundaries

In a TypeScript backend, one possible feature-oriented structure is:

src/
├── modules/
│   └── notifications/
│       ├── application/
│       │   ├── ports/
│       │   │   └── message-publisher.ts
│       │   └── use-cases/
│       │       └── request-notification.ts
│       ├── domain/
│       │   └── notification.ts
│       └── infrastructure/
│           ├── messaging/
│           │   ├── bullmq-message-publisher.ts
│           │   └── service-bus-message-publisher.ts
│           └── consumers/
│               └── notification-consumer.ts
├── shared/
│   └── messaging/
│       ├── message-envelope.ts
│       └── message-registry.ts
└── bootstrap/
    └── messaging.ts

This is an illustrative structure rather than a mandatory folder convention.

The application layer owns the port it needs. Infrastructure adapters implement that port. The bootstrap layer selects and configures the implementation.

For example, dependency injection can bind the application interface to a BullMQ-backed implementation in one deployment and an Azure Service Bus-backed implementation in another.

The consumer's business handler can remain independent of the transport, while a provider-specific consumer adapter handles delivery, settlement, retries, and error translation.

Message schemas should also be validated at the system boundary. TypeScript types disappear at runtime, and a message may have been produced by an older application version or another service.

A typed interface alone is not sufficient to guarantee that a received message has a valid payload.

Testing the boundary

A messaging abstraction also makes application services easier to test.

A unit test can replace the real message publisher with a fake implementation that records published messages.

The test can then verify that the application requests the expected operation without requiring Redis, Azure credentials, or a running worker.

Separate integration tests should verify that each adapter correctly maps application contracts to provider behavior.

For example, adapter tests can cover:

Successful publication.
Transient infrastructure failures.
Invalid or incompatible messages.
Retry and dead-letter behavior.
Duplicate delivery.
Serialization and schema validation.
Connection loss and recovery.

The goal is to test both the application-level intent and the infrastructure-level guarantees.

A passing unit test does not prove that a queue configuration is reliable in production. Likewise, an integration test against one provider does not establish that another provider has identical semantics.

Final thought

The goal is not to create a generic "queue wrapper."

The goal is to create a meaningful application-level messaging boundary while keeping infrastructure replaceable.

That boundary should protect business logic from provider-specific APIs without hiding the operational realities of asynchronous processing.

Retries still require careful policies. Consumers still need idempotency. Critical database changes may require an outbox. Delivery guarantees must be understood rather than assumed.

A well-designed abstraction does not make these problems disappear.

It gives the application a clear place to define its requirements and gives the infrastructure a clear responsibility for implementing them.

That is the difference between simply hiding a queue client behind an interface and designing a backend that can evolve its messaging infrastructure without unnecessarily rewriting its business logic.

Redis Is More Than a Cache

Tope Taiwo — Tue, 20 Oct 2026 09:07:54 +0000

Explore Redis for caching, rate limiting, sessions, coordination, messaging, and background jobs, with attention to trade-offs and failure modes

Redis is frequently introduced to developers as a caching system.

That is only one of its uses.

In backend architecture, Redis can participate in several different patterns, including caching, session management, rate limiting, distributed coordination, real-time messaging, and background-job infrastructure.

Its usefulness comes from more than simply storing data in memory. Redis provides data structures and operations that can help applications solve specific problems without repeatedly querying a database or performing expensive computations.

However, using Redis effectively requires understanding what each pattern is responsible for, what guarantees it provides, and what happens when things fail.

Cache and queue are different responsibilities

A cache answers:

Can I retrieve this expensive result without recomputing it?

A queue answers:

Can I defer this piece of work and process it asynchronously?

They are different problems.

A technology can support both without making the concepts identical.

For example, an application might use Redis to cache a user's profile while also using BullMQ to schedule email delivery, report generation, or other background tasks.

The cached profile is optimized for repeated reads. The queued job represents work that must be executed, retried, or scheduled according to the queue's processing rules.

These responsibilities should remain separate in the application architecture, even when they share the same Redis infrastructure.

Read-through caching

A useful cache pattern is:

Request
  │
  ▼
Cache
  │
  ├── Hit → Return
  │
  └── Miss
        │
        ▼
     Database
        │
        ▼
      Cache
        │
        ▼
      Return

In a read-through-style flow, the application retrieves data from the cache first. When the requested data is missing, it reads from the database and populates the cache.

Strictly speaking, this is often implemented as a cache-aside pattern: the application explicitly manages cache reads, database reads, and cache population.

For example, a NestJS service might use Redis to cache frequently accessed organizational settings, product catalog entries, or permission metadata.

async function getOrganizationSettings(
  organizationId: string,
): Promise<OrganizationSettings> {
  const key = `organization:${organizationId}:settings`;

  const cached = await redis.get(key);

  if (cached !== null) {
    return JSON.parse(cached) as OrganizationSettings;
  }

  const settings = await organizationRepository.getSettings(organizationId);

  await redis.set(key, JSON.stringify(settings), {
    EX: 300,
  });

  return settings;
}

This example uses a five-minute expiration.

The actual implementation should also validate cached data, handle serialization failures appropriately, and decide what happens when Redis is unavailable.

The cache should improve the application's performance without becoming an accidental prerequisite for retrieving data that remains available in the primary database.

Cache invalidation is the difficult part

Adding a cache is easy.

Keeping cached data correct is harder.

Questions include:

When does the cache expire?
When is it invalidated?
What happens after an update?
What happens if the database succeeds but cache invalidation fails?
Can stale data be tolerated?

These are architectural questions.

Consider an organization whose approval limit changes from ₦500,000 to ₦1,000,000.

If the database is updated but Redis still contains the previous limit, the application may continue making decisions based on outdated information.

For display-oriented data, a short period of staleness may be acceptable. For authorization, financial controls, or approval decisions, it may not be.

Common strategies include:

TTL-based expiration: Allow entries to expire after a defined period.
Explicit invalidation: Delete or update affected cache entries after a successful database write.
Versioned cache keys: Include a version in the key to prevent old entries from being reused.
Write-through approaches: Update the database and cache through a coordinated application strategy.
Event-driven invalidation: Publish changes and invalidate related entries through a reliable event-processing mechanism.

None of these strategies eliminates every failure mode.

For example, if a database transaction commits successfully but the application crashes before invalidating Redis, the cache can remain stale.

For important workflows, an outbox pattern can help: record the business change and an event in the same database transaction, then process that event asynchronously to invalidate or refresh the cache.

The key lesson is that cache consistency must be designed, not assumed.

Redis data structures are part of the value

Redis is not limited to storing serialized JSON strings.

It supports data structures with operations suited to different access patterns.

Data structure	Useful applications
Strings	Cache entries, counters, temporary tokens
Hashes	Compact representations of objects and fields
Sets	Unique membership, feature assignments, membership checks
Sorted sets	Leaderboards, priority ordering, scheduled work indexes
Lists	Simple ordered collections and queue-like operations
Streams	Append-only event records and consumer groups
Bitmaps	Compact tracking of binary states
HyperLogLogs	Approximate distinct-count analytics

The important distinction is between choosing Redis because it is available and choosing a Redis data structure because its operations match the problem.

For example, a sorted set can support a leaderboard without repeatedly sorting an entire dataset in application memory.

A set can efficiently track membership, such as which feature flags have been assigned to a particular group.

A counter can support lightweight operational metrics, subject to appropriate expiration, cardinality, and retention policies.

The data structure should follow the access pattern and consistency requirements.

Rate limiting

Rate limiting is another common use of Redis.

An API may need to restrict how frequently a client can make requests within a given period.

For example:

Incoming Request
      │
      ▼
Identify Client
      │
      ▼
Check Rate Limit in Redis
      │
      ├── Within Limit → Continue
      │
      └── Limit Exceeded → Reject

A Redis-backed limiter can maintain counters shared across multiple application instances.

Without shared state, an application running on four replicas could enforce four independent limits instead of one limit across the entire deployment.

Redis operations such as INCR and expiration can implement simple fixed-window limits. However, the counter update and expiration setup must be designed to avoid race conditions, and a fixed window can allow bursts around window boundaries.

Depending on the requirements, a sliding-window or token-bucket algorithm may be more appropriate.

Rate limiting is useful for:

Login and password-reset endpoints.
Public APIs.
OTP and verification-code requests.
Expensive report-generation endpoints.
Protection against accidental request bursts.

The key design questions include the limiting identity, the time window, the permitted burst size, and the behavior when Redis becomes unavailable.

A global rate limit, a per-user limit, and a per-IP limit are different policies. They should not be treated as interchangeable.

Session management and temporary state

Redis can also store temporary application state.

Examples include:

Server-side sessions.
Short-lived verification challenges.
Temporary workflow state.
One-time application tokens.
Expiring reservations or locks.

This is particularly useful when an application runs across multiple instances.

If a session exists only in the memory of one Node.js process, a subsequent request routed to another instance may not find it.

Shared session storage allows instances to retrieve the same session state.

However, not every token should automatically be stored in Redis. The application must define expiration, revocation, access controls, and the consequences of Redis data loss.

For example, a password-reset challenge should expire and should not remain valid indefinitely simply because its key was never explicitly deleted.

Temporary state is still security-relevant state.

Distributed locks and coordination

Suppose several application instances attempt to generate the same monthly report simultaneously.

Or imagine multiple workers trying to perform the same scheduled reconciliation.

Redis can help coordinate such work through carefully designed locking or deduplication mechanisms.

A simplified lock acquisition might look like:

SET lock:monthly-report token NX PX 30000

Here:

NX requests creation only if the key does not already exist.
PX 30000 gives the lock a 30-second expiration.
token should be a unique ownership value.

The owner must release the lock only if the stored token still matches its own token. Otherwise, an expired lock could be acquired by another worker, and the original worker could accidentally delete the new owner's lock.

A simple lock implementation is not automatically a complete distributed-locking solution.

Long-running work, process pauses, network partitions, lock expiry, and delayed operations can all create concurrency problems.

For operations where duplicate execution could cause serious damage, consider database constraints, idempotency keys, transactional guarantees, or fencing tokens in addition to coordination.

A lock can reduce contention, but it should not replace the underlying correctness guarantees of the business operation.

Background jobs with BullMQ

Redis can also support queue infrastructure such as BullMQ.

That creates a useful separation:

Application
    │
    ├── Cache
    │
    └── Job Queue
           │
           ▼
        Workers
           │
           ├── Email delivery
           ├── Report generation
           ├── File processing
           └── Notifications

But the application should not confuse those responsibilities.

Imagine an API endpoint that accepts a request to generate a large report.

Generating the report inside the HTTP request can increase response times and consume application resources while the client waits.

Instead, the API can validate the request, persist the necessary business state, enqueue a job, and return a job identifier.

A worker processes the job independently.

BullMQ supports capabilities such as delayed jobs, retries, repeatable scheduling, concurrency controls, and job prioritization.

These capabilities are useful, but they do not mean every queued job will execute exactly once.

A worker may complete an external side effect and crash before acknowledging the job. The queue may then retry it.

Consequently, a production job handler should be designed to tolerate repeated execution.

For example, an invoice-notification job should check whether the relevant notification has already been recorded or use an appropriate idempotency mechanism before performing a non-repeatable side effect.

Similarly, retry policies should distinguish transient failures from permanent failures. Repeatedly retrying an invalid request does not make it valid.

The queue is responsible for managing work. The application is responsible for making that work safe to execute.

Pub/Sub and Redis Streams

Redis can also support communication between application components.

However, Pub/Sub and Streams serve different purposes.

Redis Pub/Sub allows publishers to send messages to subscribers. It is useful for transient notifications, but messages are not retained for subscribers that are disconnected when the message is published.

For example, an application could publish a notification that a dashboard should refresh its data.

Redis Streams provide a retained, ordered stream of entries, with consumer-group capabilities for distributing processing across consumers and tracking pending messages.

Streams can be useful for event-processing workflows that require consumers to resume work after interruptions.

Even then, retention, acknowledgment, pending-message recovery, consumer coordination, and duplicate handling must be designed explicitly.

Neither Pub/Sub nor Streams should automatically be treated as a substitute for a durable business record in a relational database.

The appropriate choice depends on whether the application needs transient notification, retained event processing, or a durable source of business truth.

Redis is not automatically the source of truth

One of the most important architectural decisions is determining which system owns the authoritative version of a piece of data.

For many business applications, PostgreSQL or another primary database remains responsible for durable business records, constraints, and transactions.

Redis can then support performance, coordination, temporary state, and asynchronous processing.

For example:

PostgreSQL
    │
    ├── Authoritative business records
    ├── Transactional constraints
    └── Durable workflow state

Redis
    │
    ├── Cached reads
    ├── Rate-limit counters
    ├── Temporary state
    └── Queue and coordination infrastructure

This is an architectural choice, not a universal rule that Redis can never be a primary datastore.

Redis supports persistence and other durability-related features. However, its actual recovery guarantees depend on the configuration, persistence mode, replication, failover behavior, and operational setup.

If losing a piece of data would violate a business requirement, its storage and recovery strategy must be evaluated accordingly.

A cache entry can usually be rebuilt.

An unrecorded payment, an unprocessed business obligation, or a lost approval decision may have much more serious consequences.

Those data categories should not receive the same reliability assumptions.

Redis introduces operational trade-offs

Redis can improve application performance, but it also introduces infrastructure and operational costs.

Memory management

Redis is commonly used as an in-memory datastore. Memory is therefore a resource that must be monitored and managed.

Applications should consider:

Memory limits and eviction policies.
Key expiration and retention.
Large values and expensive data structures.
Unexpected growth in the number of keys.
Cache churn and eviction rates.

An eviction policy that works well for a disposable cache may be inappropriate for data that the application expects to retain.

Connection management

Creating a new Redis connection for every request is usually a poor approach.

Node.js applications should generally reuse managed connections, configure sensible timeouts, and handle reconnection behavior deliberately.

Connection pooling or multiplexing should follow the behavior of the selected Redis client and workload.

Redis outages

A cache outage and a queue outage have different consequences.

If Redis is used only as a cache, the application might temporarily fall back to the database, provided that doing so will not overwhelm it.

If Redis also backs background-job infrastructure, an outage may prevent jobs from being enqueued or processed.

A fallback strategy must account for these differences.

For example, sending every cache miss directly to PostgreSQL during a Redis outage can create a database traffic spike. A controlled fallback, load shedding, or temporary rejection of nonessential requests may be safer.

Likewise, silently discarding a job because Redis is unavailable may violate the business contract of the endpoint.

Resilience begins with identifying which operations can fail open, which must fail closed, and which require durable recovery.

Monitoring

A production deployment should monitor Redis as an infrastructure dependency.

Useful signals include:

Memory consumption and evictions.
Command latency and slow operations.
Connection counts and rejected connections.
Cache hit and miss rates.
Queue depth and oldest-job age.
Failed, stalled, and repeatedly retried jobs.
Replication health and persistence-related failures where applicable.

These metrics help distinguish an application problem from an infrastructure problem.

A cache hit rate alone is not sufficient evidence that Redis is improving the system. The cache must also reduce meaningful latency or database load without introducing unacceptable staleness or operational complexity.

A practical rule for backend engineers

Before introducing Redis into a feature, answer these questions:

What problem are we solving? Is it repeated computation, request throttling, coordination, temporary state, or deferred work?
What is the source of truth? Can the data be reconstructed, or must it survive failure?
What consistency guarantees are required? Can the application tolerate stale data or duplicate processing?
What happens when Redis is unavailable? Can the operation safely fall back, wait, retry, or fail?
How will the feature be monitored? Which metrics will show that the design is working?
Is Redis necessary? Would a database constraint, an existing queue, or a simpler application-level solution solve the problem with fewer moving parts?

The goal is not to use Redis in as many places as possible.

It is to use it where its data structures and operational characteristics provide a clear advantage.

Final thought

Redis is a useful infrastructure component.

The important engineering skill is not knowing that Redis exists.

It is understanding which problem Redis is solving in a particular part of the architecture.

A cache should have an explicit consistency strategy. A queue should have a retry and idempotency strategy. A rate limiter should have a defined policy. A distributed lock should have clear ownership and failure semantics.

Those decisions are what separate simply adding Redis to a backend from designing a backend that uses Redis responsibly.

Why Choose RESTApi for Notifications?

Tope Taiwo — Mon, 19 Oct 2026 05:00:00 +0000

Choose REST, polling, SSE, or WebSockets for notifications based on delivery needs, while keeping notification processing reliable and asynchronous.

Not every notification system needs WebSockets.

That is one of the architectural decisions I have been thinking through while designing backend communication systems.

Real-time communication has its place, but introducing it by default can add complexity without necessarily improving the user experience. The better approach is to understand the requirements first and select the communication model that satisfies them.

Start with the requirement

Before selecting WebSockets, Server-Sent Events (SSE), or polling, we should ask:

What does the client actually need?

If the requirement is to:

Retrieve notifications.
Mark notifications as read or unread.
Manage notification preferences.
Retrieve notification data when the client requests it.
View notifications when opening a page or refreshing a list.

REST can be sufficient.

An HTTP-based API provides a straightforward contract for retrieving notifications, updating their read status, and managing preferences. It also fits naturally into applications that already use REST for their other backend interactions.

The important point is that retrieving a notification and receiving it immediately are different requirements.

If users only need to see their notifications when they open a page, refresh a list, or make a scheduled request, persistent connections may not be necessary.

However, if the application must deliver new notifications to an already-open client with minimal delay, polling alone may not be enough. That requirement should influence the design.

Why avoid unnecessary real-time complexity?

Real-time communication introduces additional concerns:

Persistent connections and their lifecycle.
Reconnection and recovery after network interruptions.
Connection management across multiple application instances.
Load balancing and connection distribution.
Connection state and resource consumption.
Authentication and authorization throughout the connection lifecycle.
Observability for connection failures and delivery delays.

These problems can be justified when the product requires real-time updates.

But they should be driven by requirements, not by the assumption that a more sophisticated communication mechanism is automatically a better one.

For example, WebSockets may be appropriate for collaborative applications, live operational dashboards, chat systems, or workflows where clients need frequent bidirectional communication.

For a conventional notification centre, however, an HTTP endpoint may provide everything the client needs.

A simpler communication model can reduce operational overhead, make the API easier to test, and allow the team to focus on notification correctness rather than connection management.

That does not mean REST is always cheaper or better. It means the additional complexity of real-time infrastructure should deliver a measurable benefit.

Notifications can be asynchronous internally

Choosing REST for the client-facing API does not mean notification generation has to be synchronous.

This distinction is important.

A business operation should not necessarily have to wait for an email, SMS, push notification, or in-app notification to be generated before it can complete.

The architecture can be:

Business Event
      │
      ▼
Message / Job
      │
      ▼
Notification Worker
      │
      ▼
Notification Store
      │
      ▼
REST API
      │
      ▼
Client

For example, when a requisition is approved, the business operation can record the approval and initiate the notification workflow. A background worker can then process the notification and persist the resulting in-app notification.

The client retrieves that notification through REST.

This separates three concerns:

Business processing: The operation that causes the notification.
Notification processing: The creation and handling of the notification, potentially through a background job.
Notification retrieval: The API through which the client accesses the notification.

Each concern can evolve independently, provided the boundaries and contracts remain clear.

There is also an important reliability consideration: if the business transaction commits but the notification job is never published, the user may not receive the expected notification. For workflows where this matters, a transactional outbox can help ensure that the intent to publish a notification is recorded atomically with the business change.

Background processing also requires appropriate retry policies, idempotency, and failure monitoring. A job should not create duplicate notifications every time a worker retries an operation.

REST simplifies the client-facing communication boundary; it does not remove the need for reliable asynchronous processing.

Preferences belong in the design

A notification system also needs preferences.

Different users or identities may have different preferences for:

Email.
SMS.
Push notifications.
In-app notifications.

Those decisions should be represented explicitly rather than hard-coded into business workflows.

For example, a business service should not need to know whether a particular user wants email notifications or push notifications. It should communicate the relevant business event or notification intent to the notification system, which can apply the appropriate preferences and delivery rules.

Preferences may also depend on notification category, urgency, organization policy, or whether a channel is available and supported.

However, preferences should not be confused with mandatory operational messages. Security alerts, compliance notices, and other critical communications may require different rules from ordinary informational notifications.

Separating these concerns makes the system easier to maintain and allows notification channels to evolve without forcing changes throughout the business domain.

What if real-time delivery becomes necessary?

Choosing REST today does not mean ruling out real-time delivery permanently.

The system can begin with a REST API for notification retrieval and introduce polling, SSE, or WebSockets later if actual product requirements justify the additional complexity.

Each option has different characteristics:

REST with polling: The client periodically requests updates. It is straightforward, but frequent polling can generate unnecessary requests.
Server-Sent Events (SSE): The server streams updates to the client over a persistent, one-way connection. It can suit notification feeds where the server needs to push updates and the client does not need bidirectional messaging over the same connection.
WebSockets: The client and server maintain a bidirectional communication channel. This is useful when both sides need to exchange frequent real-time messages.

Regardless of the delivery mechanism, the persisted notification record should remain the source of truth. A real-time event can tell the client that new data is available, but the client should not have to rely on a transient connection to recover notifications it missed while offline.

That separation also makes it easier to support multiple client types, including web and mobile applications, without tying notification history to a particular connection.

Final thought

Technology selection should follow requirements.

If real-time delivery is necessary, real-time infrastructure may be justified.

If it is not, REST can provide a simpler and more maintainable communication boundary.

The goal is not to choose the technology with the most capabilities. It is to choose the simplest architecture that satisfies the requirements reliably, while leaving room for the system to evolve when those requirements change.

What a Failed Import Taught Me About Long-Running Data Operations

Tope Taiwo — Sun, 18 Oct 2026 05:00:00 +0000

Learn how batching, durable progress tracking, idempotency, and targeted retries make long-running data imports recoverable after failures.

When seeding initial data into a database, large data imports can be deceptively simple.

The first implementation often looks like:

Read records
   ↓
Insert records
   ↓
Done

That model changes when an operation involves thousands of records and a network failure occurs.

The challenge is no longer just inserting data. It is ensuring that the operation can recover safely when something interrupts its progress.

The problem with treating an import as one operation

If an import processes thousands of records inside one enormous transaction, a failure can make recovery difficult. The transaction may hold locks for too long, consume substantial database resources, or exceed configured timeouts.

If it processes records individually, performance may suffer because of repeated database round trips and transaction overhead.

A practical approach is batching.

Records
 │
 ├── Batch 1
 ├── Batch 2
 ├── Batch 3
 ├── ...
 └── Batch N

Instead of treating the entire dataset as one indivisible operation, the system processes manageable groups of records.

Each batch can be validated, written, and committed independently, depending on the consistency requirements of the import.

Batch size matters. Smaller batches generally reduce transaction duration and the amount of work that must be retried, but they can increase the number of database round trips. Larger batches may improve throughput, but they can consume more memory and take longer to process.

The right batch size depends on the database, record complexity, constraints, indexes, available resources, and acceptable recovery cost.

Batching creates a recovery point

Suppose:

Batch 1  → committed
Batch 2  → committed
...
Batch 47 → committed
Batch 48 → failed

We should not necessarily restart from the beginning.

If the earlier batches have been committed successfully, the system should be able to identify that progress and resume from the appropriate point.

However, batching alone does not guarantee resumability.

The application needs a reliable way to determine which records or batches have completed. A counter held only in application memory is insufficient because the process may restart and lose that information.

Depending on the import, progress can be tracked through:

A durable checkpoint recording the last successfully processed batch.
An import-job record containing the current status and progress.
Stable source identifiers that allow the application to reconcile existing records.
A combination of checkpoints and database state.

The checkpoint must accurately reflect committed work. If the system records a batch as complete before its database transaction commits, a failure could cause records to be skipped during recovery.

Conversely, if the database commits but the process crashes before updating the checkpoint, the batch may run again. That is why safe retries and idempotency remain important even when checkpoints are available.

The real problem is resumability

A long-running operation should answer:

What has completed?
What failed?
Where did the failure occur?
Can the operation safely resume?
Are records idempotent?
Can a partially completed batch be retried?
Can the system distinguish a failed operation from one that is still running?

These questions are more important than simply making the initial import work.

For a production import, I would also want to know the total number of records, how many have succeeded or failed, when the operation last made progress, and whether the failure is transient or requires intervention.

That information makes the operation observable and gives an operator a way to understand its state without reconstructing everything from application logs.

A useful design is to represent the import as a durable job with explicit states, such as:

PENDING
   ↓
RUNNING
   ├──→ COMPLETED
   ├──→ FAILED
   └──→ PAUSED / RETRYABLE

The exact states depend on the requirements. What matters is that the system can explain its current state and recover from interruption without relying on assumptions about what happened before the failure.

Network failure is normal

During one of the data-import workflows I worked through, a network connectivity failure occurred after many records had already been committed.

The useful lesson was not the specific error.

It was the architecture of the recovery problem.

A network failure does not necessarily tell us whether the database operation failed. In some situations, the client loses its connection after the database has committed a transaction but before the client receives confirmation.

The application may therefore be uncertain about the outcome.

Blindly repeating the operation can create duplicates or violate constraints. Assuming success can leave records missing.

This is why recovery logic should rely on durable database state, stable record identities, and safe retry behavior rather than treating every connection error as proof that nothing was committed.

A long-running operation must be designed with interruption in mind.

Idempotency

If we retry a batch, we should avoid creating duplicates or corrupting existing data.

That means the import needs a stable identity for each record or another mechanism for safely determining whether the record has already been processed.

For example, an imported personnel record might have a unique service number, while an organizational record might have a stable directory code. These identifiers can help the application reconcile incoming data with existing records.

In PostgreSQL, unique constraints provide an important database-level safeguard. Depending on the intended behavior, the import can use an insert, an upsert, or an explicit reconciliation step.

However, idempotency is not simply a matter of ignoring duplicate-key errors. The system must define what should happen when an existing record is encountered:

Should the existing record remain unchanged?
Should selected fields be updated?
Should conflicting values cause the import to fail?
Should the conflict be recorded for manual review?

The answer depends on whether the import is intended to create new records, synchronize an external source, or update an existing dataset.

The application and database should enforce the chosen behavior consistently.

Retrying the right unit of work

Another important consideration is deciding what to retry.

If a batch fails, retrying the entire import may waste time and repeat completed work. Retrying only the failed batch is often more efficient, provided the batch can be executed safely more than once.

A practical retry strategy should distinguish transient failures, such as temporary network interruptions or database unavailability, from permanent failures, such as invalid data or violated business rules.

Transient failures may justify a limited number of retries with exponential backoff and jitter. Permanent failures usually require correction rather than repeated execution.

For partially processed batches, the design must also establish the unit of atomicity. If each batch is committed as one transaction, a failed transaction should roll back its database changes. If records are committed individually, the retry mechanism must account for the records that succeeded before the failure.

This distinction affects how checkpoints are recorded and how much work recovery may need to repeat.

Final thought

A production data import is not just an insertion script.

It is a recoverable workflow.

Batching helps control the amount of work performed at once. Durable progress tracking makes recovery possible. Idempotency makes retries safer. Observability makes failures understandable.

Once operations become long-running, failure and resumption need to be considered part of the design rather than exceptional afterthoughts.

Designing Audit Trails for Systems That Matter

Tope Taiwo — Thu, 15 Oct 2026 05:00:00 +0000

Design audit trails that record who did what, when, and to which resource, supporting accountability in business-critical systems.

An audit log is not simply a table containing timestamps.

In systems involving inventory, approvals, assets and organizational responsibilities, knowing that something happened is not enough.

We also need to know what happened and who initiated it.

Auditability changes how a system is designed

Consider a requisition.

A useful audit trail may need to answer:

Who?
What?
When?
Which resource?
Which action?
What changed?

That is fundamentally different from simply logging:

Request completed successfully.

Audit and application logs are different

Application logs help us as engineers understand system behavior.

Audit records serve a different purpose.

For example:

Application log:
Database connection failed.

Audit record:
User approved requisition X.

The first helps operate the system.

The second records a business action.

Audit sensitive workflows

The NIS Store MIS contains workflows involving:

requisitions;
approvals;
inventory;
assets;
dispatch.

These are precisely the kinds of operations where traceability becomes important.

Do not record meaningless noise

Auditing everything can produce an enormous amount of information without creating useful accountability.

The better question is:

Which business actions must remain traceable?

Examples include:

creation;
approval;
rejection;
dispatch;
assignment;
status changes;
sensitive administrative changes.

Audit records should be difficult to misinterpret

An audit record should provide enough context to understand the action without requiring the reader to reconstruct the entire request from unrelated application logs.

This is especially important when an organization needs to investigate what happened later.

Validate audits records before persistence

It is germane to mention this because paying attention to details ensure systems are safer from attackers.

I mean Audit Input Validation (AIV).

Validating audits input before persisting in the database can help reduce attack vectors on core infrastructure especially the DATABASE.

Final thought

Auditability is not something that should be bolted onto an enterprise system at the end.

It should be considered when designing important business workflows.

If an operation matters to the organization, the system should make its history understandable.

RBAC Is Not Enough: Thinking About Authorization in Enterprise Systems

Tope Taiwo — Tue, 13 Oct 2026 06:21:18 +0000

Roles are one of the first authorization mechanisms most backend developers encounter.

For example:

admin
manager
user

The server checks the user's role and allows or rejects an operation.

That is useful.

But enterprise systems quickly demonstrate that roles alone are often insufficient.

The NIS Store MIS example

The system I have been working on above includes roles representing different responsibilities.

These include:

system administrator;
operational administrator;
store administrator;
multi-layered approving authority
store manager;
store officer;
department officer;
request officer;
dispatch officer;
auditor.

The important observation is that a person's role does not necessarily tell us everything about what they can access.

Role versus context

Consider two users with the same role.

They may still belong to different offices.

Therefore:

Same role
+
Different office
=
Potentially different access

This is where authorization becomes contextual.

RBAC

Role-Based Access Control answers questions such as:

Does this user have the store_manager role?

That is useful for coarse-grained authorization.

For example:

store_manager → approve requisition

But the system may also need to ask (which it actually does):

Is this requisition within the organizational scope this user is allowed to manage? (since it is a multi-tenant system)

That is no longer just role checking.

Authorization should be explicit

A useful authorization decision can be thought of as:

Identity
   +
Role
   +
Resource
   +
Action
   +
Context
   =
Authorization Decision

This gives us a more realistic model.

Avoid scattering authorization

One of the problems with poorly designed systems is authorization logic appearing everywhere:

if (user.role === "admin") ...

inside controllers, services and repositories.

That becomes difficult to audit.

Authorization should instead have recognizable policy boundaries.

Final thought

RBAC remains valuable.

The lesson is not to abandon it.

The lesson is to recognize when the system has moved beyond simple role-based decisions.

Enterprise authorization often needs identity, role, resource and organizational context to make a correct decision.

Sessions, Devices and JWTs in a Multi-Client Application

Tope Taiwo — Mon, 12 Oct 2026 07:00:00 +0000

Design production session management for JWT applications, including device tracking, session revocation, and web and mobile clients could be tricky.

JWT authentication is easy to demonstrate.

Production session management is harder.

A token can tell the server who a user is, but a production authentication system also needs to answer:

Which device created the session?
How many active sessions should a user have?
Can one session be revoked?
What happens after logout?
How does a mobile application behave differently from a browser?
How can suspicious sessions be identified?

These are session-management questions.

Treat sessions as first-class data

Instead of thinking only about a JWT, I think about:

User
 │
 ├── Session A
 ├── Session B
 └── Session C

A session can carry information such as:

identity;
device identifier;
creation time;
last activity;
expiration;
revocation state.

This creates a much more manageable model.

Limiting sessions

One design I have worked with uses a maximum number of active sessions per identity.

The important principle is that session limits should be enforced deliberately rather than accidentally.

When a new session is created, the application can determine whether existing sessions must be invalidated.

Device identity

For mobile applications, a device identifier can help distinguish sessions.

This does not make the device automatically trustworthy.

It simply gives the system another piece of information for session management.

The backend can then reason about:

User
 ├── Device A
 ├── Device B
 └── Device C

Revocation matters

A purely stateless JWT design can make immediate revocation difficult.

If the server issues a long-lived token and has no server-side session state, invalidating one token becomes harder.

This is why session design often combines:

short-lived access credentials;
refresh/session state;
server-side revocation.

The exact implementation depends on the application's threat model.

Mobile authentication

Mobile applications also introduce concerns such as:

secure local storage;
biometric authentication;
token refresh;
device lifecycle.

In the systems I have been designing, mobile authentication therefore cannot simply be treated as a browser authentication flow with a different UI.

Final thought

JWTs are a mechanism.

Authentication architecture is a system.

Once sessions, devices, revocation and multiple clients enter the picture, the design needs to go beyond simply generating a token.

Building Production Backends with NestJS

Tope Taiwo — Fri, 09 Oct 2026 10:05:09 +0000

My focus for this month of October will be on NestJS framework and how it has imparted my development experience as a fullstack software engineer. NestJS provides a structured framework for building Node.js applications, but the framework alone does not determine whether an application will be maintainable in production.

The architecture surrounding the framework matters a lot. I will be sharing overview on my opinions when developing a secure and scalable systems in NestJS.

Feature-oriented modules

I prefer organizing backend systems around business capabilities rather than technical file types.

For example:

src/
├── modules/
│   ├── users/
│   ├── authentication/
│   ├── payments/
│   └── notifications/
│
├── infrastructure/
├── shared/
└── main.ts

This keeps related business functionality together.

Dependency boundaries

Business logic should not directly depend on infrastructure details.

For example, application code should depend on an abstraction:

export interface EmailSender {
  send(message: EmailMessage): Promise<void>;
}

rather than directly depending on a specific provider.

The infrastructure layer can then implement it:

export class ResendEmailSender implements EmailSender {
  async send(message: EmailMessage): Promise<void> {
    // provider implementation
  }
}

This makes it possible to replace infrastructure without rewriting business logic.

The goal

The goal is not to create the largest possible architecture.

The goal is to create a system where important decisions remain easy to change.

That principle becomes increasingly important as an application grows.

Feature-Based Architecture in Node.js

Tope Taiwo — Fri, 09 Oct 2026 07:00:00 +0000

Compare feature-based and layer-based Node.js project structures, and see how organizing code by business capability supports growth.

In my previous posts, I discussed why architecture should be more important than the framework and why infrastructure should not control business logic.

The next question is practical:

How should the codebase itself be organized?

For the systems I have been working on, I increasingly prefer organizing code around business capabilities rather than putting every controller, service, repository and model into global folders.

This is what I mean by feature-based architecture.

The traditional structure

A common Node.js project starts like this:

src/
├── controllers/
├── services/
├── repositories/
├── models/
├── validators/
└── utils/

This works for small applications.

As the application grows, however, finding everything related to one business capability becomes harder.

A user-related change may involve:

controllers/UserController.ts
services/UserService.ts
repositories/UserRepository.ts
models/User.ts
validators/UserValidator.ts

The files are separated by technical role rather than business responsibility.

Organizing around features

An alternative is:

src/
├── modules/
│   ├── users/
│   ├── posts/
│   ├── orders/
│   └── notifications/

Each feature can contain its own:

domain/
application/
infrastructure/
presentation/

This is particularly useful for larger systems.

In the NIS Store MIS work, for instance, the system has distinct business areas such as:

identity and access management;
requisitions;
approvals;
assets;
inventory;
dispatch;
organizations;
notifications;
auditing.

These are not merely technical categories.

They represent different areas of the business.

A feature is a boundary

A useful feature boundary answers:

What business capability does this module own?

For example:

requisitions/
├── domain/
├── application/
├── infrastructure/
└── presentation/

The requisition module should own requisition-specific behavior.

The approval module should own approval-specific behavior.

This makes the architecture easier to reason about.

Feature-based does not mean abandoning layers

Feature-based architecture and layered architecture are not opposites.

They can work together.

For example:

modules/
└── requisitions/
    ├── domain/
    ├── application/
    ├── infrastructure/
    └── presentation/

The feature provides the boundary.

The internal layers provide separation of responsibilities.

This combination is more useful to me than choosing one pattern dogmatically.

Why this matters

As systems grow, organization becomes part of architecture.

A codebase should make it obvious:

where a business rule lives;
which module owns it;
which infrastructure implements it;
where its HTTP interface is exposed;
how it can be tested.

Good folder structure cannot fix bad architecture.

But good boundaries make bad coupling easier to detect.

Final thought

The purpose of feature-based architecture is not to make the folder tree look sophisticated.

It is to make the business structure visible in the code.

When I open a production backend, I want to understand the system by understanding its capabilities.

That is the direction I am increasingly taking with my Node.js architecture.

From Silence to Sound: Fixing Audio on Ubuntu Linux on my Samsung Galaxy Book

Tope Taiwo — Sat, 02 Aug 2025 22:58:42 +0000

Over four days. That’s how long I spent trying to get the internal speakers on my Samsung Galaxy Book2 (NT950XDA) to produce a single sound after switching from Windows to Ubuntu.

And I wasn’t alone.

The Problem

Right out of the box, Ubuntu (even the latest 24.04) doesn’t play well with the Realtek ALC298 audio codec on many Samsung Galaxy Book devices. Headphones might work, but internal speakers remain silent due to uninitialized smart speaker amps.

I found myself crawling through obscure forum posts, GitHub issues, and deep dmesg logs. I came across broken workarounds, kernel parameters, even dead-end firmware downloads. Nothing stuck — until I discovered a promising thread from the Manjaro community and a early work of @joshuagrisham.

The Breakthrough

The answer turned out to be hda-verb, a tool for sending manual codec commands to initialize the amp. After copying dozens of lines of hda-verb commands into a script and running it manually, audio finally burst through my speakers.

It worked—but only until the next reboot.

The Automation

That’s when I decided to build a real solution:

Auto-run the fix at boot and after resume
Avoid breaking any existing audio setups
Use systemd services for reliability
Keep it simple, safe, and open source

The Tool: galaxybook-linux-setup

Now available on GitHub: https://github.com/topzyray/galaxybook-linux-setup

Features:

🛠 Initializes smart speaker amps via hda-verb
🌀 Runs fix on boot + resume using systemd
🎚 Disables PulseAudio's suspend-on-idle to prevent speaker freezing
📁 Backs up existing config files to avoid damage
⚙️ Detects your current user and system safely
🤝 MIT-licensed, contributions welcome

Installation is as easy as:

git clone https://github.com/topzyray/galaxybook-linux-setup.git
cd galaxybook-linux-setup
chmod +x install.sh
./install.sh

Reboot when prompted, and you’re done. No more silent boots.

Who This Might Helps

Galaxy Book2 NT950XDA users or other device with similar issue or device/driver info
Anyone with Realtek ALC298 codec and no speaker output
Linux newcomers switching from Windows
Ubuntu/Mint/Pop_OS/Manjaro users on 5.8+ kernels though I tested on 6.14.

Credit Where Due

Joshua Grisham @joshuagrisham – GitHub
Community post from user @drlucas – Manjaro Forum
Access hda-ver used here

The Bigger Picture

This project reminded me why open source matters. Without the knowledge of those before me, I’d still be rebooting into silence.

Now, I’m extending this to include fingerprint readers, screen brightness fixes, and more for Galaxy Book users and I will also be testing on other devices. Please you can test too.

NOTE: The script does not delete any file rather it create a backup whenever updating an existing file.

Have a similar issue? Found this useful? Contribute, fork, share, or open an issue!

👉 Repo: https://github.com/topzyray/galaxybook-linux-setup

If you have further question, suggestion or observations please leave them in the comment section. Thank you for reading.