DEV Community: Tope Taiwo

Payments Are a Workflow, Not Just an API Call

Tope Taiwo — Thu, 29 Oct 2026 08:54:59 +0000

"Design payment processing as a recoverable workflow with verified outcomes, idempotency, separate financial records, and reconciliation."

A payment integration can look deceptively simple.

Call a payment provider.

Receive a response.

Mark the transaction successful.

Real payment systems are more complicated.

A payment involves external infrastructure, asynchronous communication, financial records, and business rules that must remain consistent even when something fails.

A request can time out after the provider has processed it. A webhook can arrive more than once. A customer can close the browser before the checkout process finishes. A payment can be completed externally while the application is temporarily unavailable.

These are not exceptional cases that can simply be ignored. They are normal conditions in distributed systems.

The challenge is not just initiating a payment. It is determining what happened, recording the outcome reliably, and ensuring that the rest of the application responds correctly.

A payment is a workflow

A better mental model is:

Initiated
   │
   ▼
Pending
   │
   ├── Successful
   ├── Failed
   └── Abandoned

The system needs to model those states.

In practice, the lifecycle may require additional states, such as created, processing, expired, cancelled, reversed, or refunded, depending on the provider and business requirements.

It is also important to distinguish a payment attempt from an order or purchase.

An order represents what a customer intends to buy. A payment attempt represents an attempt to collect the money for that order. One order may have several payment attempts if an earlier attempt fails or expires.

Treating these as separate concepts helps prevent retries from creating duplicate purchases or losing the relationship between the customer's intention and the financial transaction.

A simplified domain model might look like this:

type PaymentStatus =
  | "pending"
  | "successful"
  | "failed"
  | "abandoned"
  | "refunded";

type Payment = {
  id: string;
  orderId: string;
  providerReference: string;
  amountMinor: bigint;
  currency: string;
  status: PaymentStatus;
};

This is illustrative rather than a complete payment schema. A production implementation would also need to define valid state transitions, timestamps, provider identifiers, and any relevant refund or reconciliation records.

The key point is that payment status should represent a verified business fact, not merely the outcome of one HTTP request.

State transitions should also be controlled. A delayed failure notification, for example, must not blindly overwrite a payment that has already been verified as successful. Similarly, a refund should not be represented as if the original payment never occurred.

Financial workflows need explicit rules for how states change and which evidence permits each transition.

Why the API response is not the entire story

External payment providers can communicate through callbacks or webhooks.

That means payment processing is distributed.

Our application and the payment provider are separate systems.

Neither system can assume that every message will arrive immediately, exactly once, or while the other system is available.

Consider a customer who initiates checkout. The provider processes the payment, but the response to our backend times out before the application receives confirmation.

The application now has an uncertain outcome.

If it immediately marks the payment as failed and starts another payment attempt, the customer could eventually be charged twice.

If it marks the payment as successful merely because the checkout request was accepted, the customer could receive a product without a completed payment.

The backend therefore needs to handle:

duplicate notifications;
delayed notifications;
verification;
inconsistent intermediate states.

It should also handle temporary provider outages, request timeouts, malformed messages, and events that arrive in an unexpected order.

A reliable design separates payment initiation from payment confirmation.

Initiating a payment means that the application has requested a payment operation. It does not necessarily mean that money has been received.

Similarly, redirecting a customer to a success page is not proof that the payment succeeded. The browser is controlled by the customer and cannot serve as the sole authority for a financial decision.

The backend needs a trusted way to establish the final payment outcome.

Verify before granting access

For a provider such as Paystack, the backend should treat the provider's verified transaction status as evidence for deciding whether to complete the business operation.

The exact verification process depends on the provider's integration model, but it commonly involves verifying the transaction through a trusted server-side API or validating an authenticated webhook and checking the relevant transaction details.

The application should confirm that the transaction matches its own expected values, including:

the provider reference;
the expected amount;
the currency;
the associated order or purchase;
the verified payment status.

The amount and currency are especially important. A successful transaction is not sufficient if it does not correspond to the amount and currency expected for the purchase.

For a webhook, signature verification must be performed according to the provider's documented requirements, using the unmodified request body where required. Invalid signatures should be rejected. Secrets should remain on the server and must not be exposed to the frontend.

A typical flow looks like this:

Customer
   │
   ▼
Backend creates payment attempt
   │
   ▼
Payment provider
   │
   ├── Checkout response
   │
   └── Webhook / verification
              │
              ▼
       Verify authenticity
              │
              ▼
       Verify transaction details
              │
              ▼
       Record payment outcome
              │
              ▼
       Apply business effects

The browser redirect and webhook can both trigger a verification attempt, but neither should cause the application to bypass its own payment-state rules.

A useful principle is to separate the evidence received from the provider from the business decision made by the application.

The provider reports a transaction outcome. The application determines whether that outcome satisfies the conditions required to fulfil the associated order.

Idempotency matters

If the same payment notification arrives more than once, processing it twice should not create two successful purchases.

The application needs a stable transaction identity and idempotent processing.

Duplicate delivery is a normal possibility in webhook-based systems. Providers may retry notifications when delivery fails or when they do not receive the expected acknowledgement.

A webhook handler should therefore be safe to execute more than once for the same event.

For example, the application can use the provider's transaction reference or event identifier to identify repeated processing attempts. Database uniqueness constraints can enforce important parts of that guarantee.

A simplified table might include:

CREATE TABLE payments (
    id UUID PRIMARY KEY,
    order_id UUID NOT NULL,
    provider_reference TEXT NOT NULL UNIQUE,
    amount_minor BIGINT NOT NULL,
    currency CHAR(3) NOT NULL,
    status TEXT NOT NULL,
    created_at TIMESTAMPTZ NOT NULL DEFAULT NOW(),
    updated_at TIMESTAMPTZ NOT NULL DEFAULT NOW()
);

This illustrates how a unique provider reference can prevent duplicate payment records for the same reference. A production schema should also enforce valid statuses and appropriate relationships, and should define whether a provider reference is unique globally or within a provider account.

However, idempotency is not simply checking whether a row already exists.

The application must also handle concurrent processing. Two webhook deliveries could arrive close together, both observe a pending payment, and both attempt to apply the same business effect.

A robust implementation uses database transactions, appropriate uniqueness constraints, conditional state transitions, or row-level locking where necessary to prevent duplicate effects.

For example, marking an order as paid and recording the corresponding payment result should be coordinated so that a crash does not leave the system with a successful payment but no durable record of the resulting business decision.

Idempotency must extend beyond the payment row itself. If successful payment grants an examination subscription, creates an entitlement, or records a referral commission, repeating the payment handler must not grant that entitlement or commission twice.

The desired behavior is not necessarily that the handler runs only once.

It is that repeated processing produces the same valid business outcome.

Keep financial records separate from business effects

A successful payment can trigger several operations:

activating a subscription;
granting access to an examination product;
confirming an order;
recording a referral commission;
scheduling a payout;
sending a receipt or notification.

These operations are related, but they are not the same thing.

The payment record captures the financial outcome. The entitlement or subscription records capture what the customer is allowed to use. Referral records capture commission eligibility and amounts. Payout records capture the movement of funds to a beneficiary.

Keeping these responsibilities distinct makes the system easier to audit and recover.

For example, if a payment is confirmed but subscription activation fails, the application should not lose the payment result simply because a downstream operation encountered an error.

The system needs a way to retry the failed business effect without charging the customer again or pretending that the payment itself failed.

This is one reason asynchronous processing and transactional outbox patterns can be useful.

The application can persist the verified payment outcome and the intent to perform downstream work in one database transaction. A worker can then process that work, using idempotent operations to handle retries safely.

This does not make every operation atomic across every system. It creates a durable path for coordinating work across system boundaries.

Payment architecture

For systems such as the education products I have worked on, payment state can affect:

access;
subscriptions;
examination products;
referrals;
payouts.

That means payment processing should not be treated as a small utility function.

It is part of the business workflow.

A practical architecture might separate responsibilities into components such as:

Payments Module
   │
   ├── Payment Initiation
   ├── Provider Integration
   ├── Webhook Verification
   ├── Payment State Management
   ├── Reconciliation
   └── Refund Handling
           │
           ▼
    Business Workflows
           │
           ├── Entitlements
           ├── Subscriptions
           ├── Referrals
           └── Payouts

The exact module boundaries depend on the application, but the separation helps prevent provider-specific behavior from spreading throughout the business domain.

For example, a subscription module should not need to know how a specific provider signs webhook requests. It should respond to a trusted payment outcome or a clearly defined application event.

Similarly, the payment module should not assume that recording a successful payment means every downstream operation has already completed.

This separation also makes it easier to support another payment provider in the future without rewriting the business rules for subscriptions, entitlements, or referrals.

The objective is not to abstract every provider detail prematurely. It is to prevent external payment infrastructure from becoming inseparable from the application's business logic.

Reconciliation is part of payment engineering

Webhooks and retries improve reliability, but they do not eliminate uncertainty.

A webhook might never be processed successfully. A temporary outage might leave a payment in a pending state. A local database update might fail after the provider has completed the transaction.

A payment system therefore needs a reconciliation process.

Reconciliation compares the application's payment records with the provider's transaction records and identifies discrepancies that require investigation or correction.

For example, a scheduled task might examine transactions that have remained pending beyond an expected period, query the provider for their current status, and update the local state when the result has been verified.

The process should be designed carefully. It should not blindly overwrite local records or repeat business effects without applying the same verification and idempotency rules used by webhook processing.

Useful operational metrics include:

the number and age of pending transactions;
webhook delivery and processing failures;
verification failures;
duplicate events;
payment-to-entitlement delays;
discrepancies discovered during reconciliation;
refund and payout failures.

These measurements help distinguish a payment provider problem from an application processing problem.

They also make it easier to detect situations where the financial transaction succeeded but the customer has not received the expected product or service.

Final thought

A payment provider gives us infrastructure.

The application still needs to model its own business state.

It must distinguish initiation from confirmation, verify the transaction, handle duplicate and delayed messages, protect against repeated business effects, and recover when part of the workflow fails.

That is why payment engineering is fundamentally a workflow-design problem.

The goal is not simply to process a successful payment.

It is to ensure that the payment, the financial record, and the business outcome remain consistent—even when the systems involved do not behave as expected.

Multi-Tenancy: What Changes When One Application Serves Many Organizations

Tope Taiwo — Wed, 28 Oct 2026 15:11:58 +0000

"Design multi-tenant applications with deliberate data isolation, trusted tenant context, tenant-aware authorization, and boundary testing."

Building software for one organization is different from building software that serves many organizations.

This distinction has become particularly relevant to me through work around school-management software.

When an application serves a single organization, many assumptions can remain implicit. Users belong to the same organizational environment. Configuration is relatively straightforward. Data access often assumes a single administrative boundary.

Those assumptions become dangerous when the same application serves multiple organizations.

A school-management platform, for example, may serve hundreds of schools, each with its own students, teachers, classes, academic sessions, billing arrangements, and administrative policies.

The application may share infrastructure, but the organizations using it must remain distinct in the ways that matter to the business.

That is where multi-tenancy becomes an architectural concern rather than just a database design decision.

What is multi-tenancy?

A multi-tenant application serves multiple organizations while sharing the application infrastructure.

Conceptually:

Application
   │
   ├── School A
   ├── School B
   └── School C

The challenge is ensuring that each tenant's data remains correctly isolated.

Sharing infrastructure can reduce operational overhead and make it easier to deploy, monitor, and maintain a common application. It can also allow improvements to be delivered consistently across organizations.

However, sharing an application does not mean sharing unrestricted access to its data.

Each tenant may have its own users, roles, configuration, records, and business rules. Some capabilities may be common to all tenants, while others may depend on the organization's subscription, policies, or enabled features.

A multi-tenant architecture must support these differences without allowing one organization's operations to interfere improperly with another's.

There are several common approaches to tenant isolation:

Shared database and shared tables: Multiple tenants use the same tables, with a tenant identifier distinguishing their records.
Shared database with separate schemas: Tenants share a database server but use separate database schemas.
Separate databases: Each tenant has a dedicated database, providing a stronger separation boundary at the database level.

These approaches involve different trade-offs in cost, operational complexity, isolation, backup and restoration, and tenant-specific customization.

A shared database with tenant-scoped tables is often a practical starting point for a SaaS application. It is not automatically the right choice for every system, particularly where contractual, regulatory, or operational requirements demand stronger isolation.

The important point is to choose an isolation strategy deliberately rather than allowing it to emerge accidentally from the initial schema.

Tenant context

A request should carry enough information for the application to determine its tenant context.

Conceptually:

Request
 │
 ▼
Identity
 │
 ▼
Tenant
 │
 ▼
Authorization
 │
 ▼
Business Operation

Tenant context should not be treated as an optional detail.

It affects authorization and data access.

A user may belong to several organizations, so identifying the authenticated user is not always sufficient to determine which organization's resources the user is permitted to access.

For example, a teacher may work at two schools. A school administrator may manage one school, while a platform administrator may have a separate, explicitly authorized level of access.

The application needs to establish the active organizational context and verify that the authenticated identity is permitted to operate within it.

A request might include a tenant identifier in its route or headers, or the application might resolve the tenant from a verified domain or subdomain. Regardless of the mechanism, the supplied identifier should be treated as a request to select a context—not as proof of authorization.

For example:

type TenantContext = {
  tenantId: string;
  userId: string;
  membershipId: string;
  roles: string[];
};

This is an illustrative representation of the context established after authentication and membership verification. It should not be constructed by blindly trusting client-supplied role values or tenant identifiers.

A reliable request flow should:

Authenticate the identity.
Resolve the requested tenant.
Verify that the identity has an active membership or other explicitly authorized access to that tenant.
Establish the tenant context for the operation.
Enforce authorization and tenant-scoped data access.
Record relevant actions in the audit trail.

This context should be propagated consistently through the application layers that need it.

If controllers enforce tenant boundaries but repositories can execute unrestricted queries, the system still has a gap. If background jobs process tenant-specific records without carrying their tenant context, the same problem can reappear outside the HTTP request lifecycle.

Tenant context is therefore a cross-cutting application concern.

Data isolation

A multi-tenant database design needs a clear strategy for separating tenant data.

A shared database can still provide logical tenant isolation, provided every relevant operation correctly applies tenant boundaries.

The dangerous situation is one where a developer can accidentally query data without tenant context.

Consider a simplified PostgreSQL table:

CREATE TABLE students (
    id UUID PRIMARY KEY,
    tenant_id UUID NOT NULL,
    full_name TEXT NOT NULL,
    created_at TIMESTAMPTZ NOT NULL DEFAULT NOW()
);

The tenant_id column identifies the organization that owns each record.

A tenant-scoped query should use that boundary explicitly:

SELECT id, full_name, created_at
FROM students
WHERE tenant_id = $1;

The value supplied to $1 must come from the application's verified tenant context, not from an untrusted assumption about the requester.

The column alone does not guarantee isolation. A query that forgets the tenant predicate can expose records belonging to other organizations.

This is why tenant isolation should be reinforced at multiple levels.

At the application layer, repositories and data-access functions should require tenant context for tenant-owned operations. Avoid APIs that make unrestricted access the easiest default.

For example, a repository method that accepts a tenant identifier makes the ownership boundary explicit:

interface StudentRepository {
  findById(tenantId: string, studentId: string): Promise<Student | null>;
}

The implementation must apply both identifiers to the query. Merely adding tenantId to a method signature does not enforce isolation if the implementation ignores it.

At the database layer, PostgreSQL Row-Level Security (RLS) can provide an additional enforcement mechanism. Policies can restrict which rows a database role may read or modify based on a tenant context established for the database transaction or connection.

RLS must be configured carefully. The application role, table ownership, policy definitions, and connection-pool behavior all matter. For example, table owners and roles with certain privileges can bypass policies unless the design explicitly accounts for that behavior. A session-level tenant setting must also not leak between pooled connections or transactions.

At the schema level, constraints should prevent invalid relationships. If a student belongs to one tenant, a relationship between that student and another tenant's class should not be accepted merely because both record identifiers exist.

Composite keys or unique constraints involving tenant_id, together with appropriate foreign-key design, can help enforce these ownership relationships.

The objective is to make tenant isolation an invariant of the system rather than a convention that every developer must remember.

Authorization must include tenant context

Authentication answers the question: who is making the request?

Authorization answers: what is that identity allowed to do?

Multi-tenancy adds another important question: within which organizational context is the action permitted?

A user may be authorized to view student records in School A but have no permission to view the same category of records in School B.

Even when both schools use the same application and database, their permissions must remain distinct.

This is particularly important for operations involving approvals, financial records, personnel information, and administrative configuration.

A useful authorization decision may depend on several factors:

the authenticated identity;
the active tenant;
the user's membership in that tenant;
the role or permissions assigned within that membership;
the resource being accessed;
the action being performed;
any applicable business rules.

A global role such as admin is often too broad to express these distinctions safely.

The application should evaluate permissions in the context of the tenant and the requested operation. Tenant-specific authorization must also apply to resource identifiers: knowing a valid student ID, invoice ID, or requisition ID should not grant access to a resource belonging to another tenant.

This is one reason authorization should not be implemented solely by hiding buttons or links in the frontend. The backend must enforce the actual access rules.

Roles can also be tenant-specific

A user may have a different role in different organizational contexts.

That means:

User
 ├── Tenant A → Role X
 └── Tenant B → Role Y

For example, a user might be an administrator at one school and a teacher at another.

Representing these permissions as one global role would create ambiguity. Should the user receive administrator privileges everywhere, or only within the school where those privileges were granted?

A tenant-aware membership model makes the distinction explicit:

User
  │
  ├── Membership A
  │      ├── Tenant A
  │      └── Role: School Administrator
  │
  └── Membership B
         ├── Tenant B
         └── Role: Teacher

A simplified relational design might include:

User
 └── TenantMembership
        ├── userId
        ├── tenantId
        └── status

TenantMembership
 └── MembershipRole
        ├── membershipId
        └── roleId

In this model, the membership establishes the user's relationship with a tenant, and roles are assigned within that relationship.

The design can be extended to support permissions, invitations, membership expiry, and role changes without treating the user's identity as equivalent to their authority in every organization.

Platform-level administration can still exist, but it should be represented as a separate, deliberate authorization path rather than an accidental consequence of tenant membership.

The key is to distinguish the identity itself from the organizational permissions associated with it.

Tenant isolation extends beyond database queries

Database queries are an important part of multi-tenancy, but they are not the only place where tenant boundaries can fail.

Consider the other components of a typical SaaS application.

Caching: A cache key that includes a resource ID but omits the tenant context may return one organization's data to another organization. Tenant-owned cache entries should include the relevant tenant scope where appropriate.

Background jobs: A queued job should retain the tenant identity required to process its work. A worker must not assume that the tenant context from an earlier HTTP request is still available.

File storage: Uploaded documents, profile images, reports, and exports need tenant-aware ownership and access checks. An unpredictable object-storage path is not a substitute for authorization.

Search and reporting: Search indexes, analytics queries, dashboards, and exports must preserve tenant boundaries. A correctly scoped API does not help if a reporting query aggregates records across organizations unintentionally.

Logs and audit trails: Tenant identifiers should be included where useful for traceability, while sensitive information should be handled according to appropriate logging and retention policies.

Configuration: Tenant-specific settings should be resolved within the active organizational context. One tenant's feature flags, branding, or business rules should not inadvertently affect another.

These concerns become especially important as the application grows. A new integration or background-processing feature can introduce a data-access path that bypasses assumptions enforced elsewhere.

Tenant isolation needs to be considered across the complete system, not just its primary database tables.

Testing tenant boundaries

Multi-tenancy also changes what needs to be tested.

Testing whether a user can retrieve their own records is not enough. We must also test whether that user is prevented from accessing another tenant's records.

Useful scenarios include:

a user cannot retrieve a record belonging to another tenant by changing its identifier;
a user cannot update or delete another tenant's resource;
a tenant administrator cannot administer a different tenant;
a user with different roles in different organizations receives the correct permissions in each context;
cache entries and background jobs preserve the correct tenant scope;
invalid or missing tenant context is rejected where required;
database constraints prevent cross-tenant relationships;
tenant-specific exports and reports contain only authorized records.

These tests should exercise the actual authorization and data-access paths, including integration tests against the database where appropriate.

Where PostgreSQL RLS is part of the security design, its policies should be tested using the same database roles and transaction patterns used by the application. Tests run only as a privileged database owner may fail to reveal problems that affect the real application role.

The goal is to verify the boundary, not merely the successful execution of ordinary operations.

Why this connects to my other work

The same architectural principles appear in enterprise systems:

identity;
organization;
authorization;
data boundaries;
auditability.

The domain may change from schools to government offices, but the architectural questions are surprisingly similar.

In both cases, an identity operates within an organizational structure, and access depends on the relationship between that identity, the organization, and the requested resource.

A school-management platform may organize data around schools, classes, students, and teachers. An enterprise or government system may organize data around departments, offices, personnel, assets, and approvals.

The domain models differ, but both systems need to establish ownership, enforce contextual permissions, preserve traceability, and prevent one organizational unit from accessing another's data without authorization.

This is why I see multi-tenancy as more than a SaaS feature. It is an exercise in making organizational boundaries explicit and enforceable.

Final thought

Multi-tenancy is not simply adding a tenantId column.

It is a system-wide data and authorization boundary.

That boundary must be reflected consistently across authentication, membership, authorization, database access, caching, background jobs, file storage, reporting, and testing.

A shared application can serve many organizations effectively, but only when the system makes it difficult for one tenant's operations to cross into another tenant's data unintentionally.

The objective is not merely to support multiple organizations.

It is to support them without compromising the isolation, integrity, and accountability each organization expects.

Building Frontends That Respect Backend Architecture

Tope Taiwo — Tue, 27 Oct 2026 10:53:00 +0000

"Apply clear frontend boundaries by separating server and client state, centralizing API access, and organizing code around features."

Although much of my recent architectural focus has been backend engineering, full-stack systems have reminded me that frontend architecture also needs boundaries.

A backend can be beautifully designed and still become difficult to work with if the frontend mixes networking, server state, local state and UI behavior indiscriminately.

When every component is responsible for fetching data, handling errors, managing loading indicators, transforming responses and maintaining shared state, the application gradually becomes harder to reason about.

Small changes begin to affect unrelated components. API changes spread across the UI. State becomes difficult to trace. Components become tightly coupled to implementation details that should have remained behind a boundary.

These are not necessarily problems with React or any particular frontend framework.

They are architectural problems.

The same principles that make backend systems easier to maintain—separation of concerns, explicit contracts and clear ownership—also apply to frontend engineering.

Separate server state from client state

This distinction is particularly important.

Data retrieved from an API is server state.

Examples include:

users;
requisitions;
inventory;
notifications.

UI state is different.

Examples include:

modal visibility;
selected filters;
temporary form state.

These two categories should not automatically be managed the same way.

Server state represents information owned by an external system, usually the backend. It may become stale, change independently of the current browser session, require refetching, and need synchronization after mutations.

Client state represents information that the frontend owns for the current interaction or application experience.

For example, a requisition list may be server state, while the currently selected status filter is client state. The filtered result may be derived from those two values rather than stored as another independent copy.

This distinction matters because every additional copy of the same data creates another opportunity for inconsistency.

If a requisition is approved, the frontend should not have to manually update several unrelated stores and component states to keep the interface synchronized.

Instead, the architecture should make it clear where the authoritative data comes from and how the UI responds when that data changes.

React Query

For server state, TanStack Query (formerly React Query) provides useful mechanisms around:

fetching;
caching;
invalidation;
mutations;
loading states;
error states.

That reduces the temptation to put every API response into a global client store.

It also provides a consistent approach to concerns that otherwise tend to be implemented repeatedly across components.

For example, a requisition query can be defined independently of the component that displays the result:

import { useQuery } from "@tanstack/react-query";

import { requisitionsApi } from "@/lib/api/requisitions";

export function useRequisitions() {
  return useQuery({
    queryKey: ["requisitions"],
    queryFn: () => requisitionsApi.list(),
  });
}

A page or component can consume the query and focus on presentation:

"use client";

import { useRequisitions } from "@/hooks/use-requisitions";

export function RequisitionList() {
  const { data: requisitions, isPending, isError } = useRequisitions();

  if (isPending) {
    return <p>Loading requisitions...</p>;
  }

  if (isError) {
    return <p>Unable to load requisitions.</p>;
  }

  return (
    <ul>
      {requisitions.map((requisition) => (
        <li key={requisition.id}>{requisition.reference}</li>
      ))}
    </ul>
  );
}

These examples assume that the API client returns data in the shape expected by the component. In a production application, that contract should be defined and validated appropriately.

Mutations should also have an explicit strategy for keeping cached data consistent. Depending on the operation, that might involve invalidating a query, updating cached data, or using an optimistic update with appropriate rollback behavior.

There is an important distinction here: caching does not make the frontend the source of truth. It provides a mechanism for reusing and synchronizing data that is ultimately owned by the backend.

Query keys, invalidation rules and mutation behavior are therefore part of the frontend's data architecture, not just implementation details.

Zustand

A lightweight client-state library such as Zustand can instead handle application state that genuinely belongs on the client.

For example, an application might need to retain sidebar visibility, a temporary multi-step workflow, or a shared selection used across several components.

Zustand can provide a simple way to manage that state without introducing unnecessary complexity.

import { create } from "zustand";

type WorkspaceState = {
  sidebarOpen: boolean;
  toggleSidebar: () => void;
};

export const useWorkspaceStore = create<WorkspaceState>((set) => ({
  sidebarOpen: true,

  toggleSidebar: () =>
    set((state) => ({
      sidebarOpen: !state.sidebarOpen,
    })),
}));

The store owns the sidebar's UI state. It does not need to own the API responses displayed inside the sidebar.

This is the architectural distinction I want to preserve.

A global store should not become the default destination for every piece of data simply because multiple components need access to it.

Before adding state to a shared store, it is worth asking whether that state actually needs to be shared, whether it can be derived, and who owns its lifecycle.

Not every state value needs a library. Local component state remains appropriate for interactions that do not need to be shared, while form libraries can handle more involved form state and validation.

The architectural distinction is more important than the particular library.

API clients

I also prefer keeping HTTP concerns behind a consistent client boundary.

For example:

UI
 │
 ▼
Query / Mutation
 │
 ▼
API Client
 │
 ▼
Backend

The UI should not need to understand every Axios configuration detail.

A centralized API client can own concerns such as:

base URLs and request configuration;
authentication and credential handling;
request and response interceptors;
timeout behavior;
consistent error normalization;
request metadata and correlation identifiers, where appropriate.

Feature-specific API modules can then expose operations in terms that make sense to the application.

For example, a requisition module might expose list(), getById() and approve(), rather than requiring every component to construct its own HTTP requests.

import { apiClient } from "@/lib/api/client";
import type { Requisition } from "@/types/requisition";

export const requisitionsApi = {
  async list(): Promise<Requisition[]> {
    const response = await apiClient.get<Requisition[]>("/requisitions");

    return response.data;
  },

  async approve(id: string): Promise<Requisition> {
    const response = await apiClient.post<Requisition>(
      `/requisitions/${id}/approve`,
    );

    return response.data;
  },
};

This example illustrates the boundary; a real implementation should use the actual response envelope and error contract defined by the backend.

The benefit is not simply fewer lines of code.

It is the ability to change HTTP configuration, error handling or transport details without rewriting every component that consumes the API.

It also gives the frontend a natural place to translate technical transport concerns into application-level behavior.

However, a shared client should not become an excuse to hide important semantics. Authentication failures, authorization failures, validation errors and transient network failures may require different responses. A generic interceptor should not silently treat them all as equivalent.

Respect the backend contract

A frontend and backend are not independent systems that happen to communicate over HTTP.

They are two parts of a system with an explicit contract.

That contract includes more than endpoint URLs. It also includes request and response shapes, validation rules, authorization behavior, pagination, error formats, and the meaning of successful operations.

For example, hiding an approval button in the UI does not enforce authorization. The backend must still verify that the current identity has permission to approve the requisition.

Similarly, frontend validation can improve usability, but the backend remains responsible for enforcing business rules.

TypeScript helps maintain consistency within the frontend codebase, but a TypeScript interface does not automatically validate data received over the network at runtime.

Where necessary, runtime schema validation and shared contract definitions can reduce the risk of frontend and backend assumptions drifting apart.

The objective is not to duplicate the entire backend domain model in the browser. It is to establish a clear, dependable contract between the two sides.

Next.js

Next.js provides another layer of architectural decisions.

With the App Router, developers need to understand:

server components;
client components;
routing;
data fetching;
loading and error boundaries.

These concepts influence where code executes, how data reaches the UI, and which responsibilities belong on the server or in the browser.

Server Components are useful for rendering content that does not require browser-side interactivity. They can fetch data on the server and keep server-only implementation details out of the client bundle.

Client Components are appropriate when a feature requires state, event handlers, effects, or browser APIs.

The distinction should be intentional.

Marking a large component tree as client-rendered simply because one small section needs interactivity can expand the client-side JavaScript unnecessarily. Conversely, forcing interactive behavior into Server Components creates avoidable friction.

Data fetching also needs deliberate ownership. Next.js server-side data access and a client-side query cache solve related but different problems.

Server Components can be appropriate for initial rendering and server-side access to data. TanStack Query is useful when the client needs ongoing query caching, refetching, mutation handling, or synchronization during an interactive session.

Using both is possible, but their responsibilities should be clear. Otherwise, the same data may be fetched and cached independently without a coherent invalidation strategy.

Loading and error boundaries deserve similar attention. They should communicate the state of the application meaningfully rather than serve as generic fallbacks that conceal the source of a failure.

Again, the framework should support the application architecture.

It should not determine the architecture by accident.

Organize around features and responsibilities

As frontend applications grow, folder structure can either reinforce or weaken these boundaries.

A feature-oriented structure can keep related API operations, query hooks, types and components close to one another:

src/
├── app/
│   ├── requisitions/
│   └── inventory/
├── components/
│   ├── ui/
│   └── layout/
├── features/
│   ├── requisitions/
│   │   ├── api/
│   │   ├── hooks/
│   │   ├── components/
│   │   ├── schemas/
│   │   └── types/
│   └── inventory/
│       ├── api/
│       ├── hooks/
│       ├── components/
│       ├── schemas/
│       └── types/
├── lib/
│   └── api/
└── stores/

This is one possible arrangement, not a universal prescription.

A small application may not need all these folders. A larger application may benefit from stronger feature boundaries and explicit shared contracts.

The underlying principle is that related responsibilities should be easy to locate, while unrelated features should not become tightly coupled through shared components or global state.

A folder structure alone cannot guarantee good architecture. The imports, ownership rules and dependencies between modules matter more than the names of the directories.

Final thought

Full-stack engineering is not about treating frontend and backend as unrelated applications.

The contracts between them are part of the architecture.

Server state and client state have different responsibilities. API clients should isolate transport concerns. Components should focus on presenting and interacting with data. Framework features should be used according to their strengths.

These boundaries reduce unnecessary coupling and make changes easier to reason about.

A clean API boundary makes both sides easier to evolve.

A clean frontend architecture helps ensure that the backend's design remains an asset rather than becoming obscured by the way the UI consumes it.

Performance Engineering: Measure Before You Optimize

Tope Taiwo — Mon, 26 Oct 2026 18:03:42 +0000

"Improve backend performance through representative benchmarks, meaningful baselines, and measurements that identify real system bottlenecks."

Performance engineering is often presented as a collection of optimization tricks.

Use caching.

Use indexes.

Use queues.

Use clustering.

Use streams.

These can all be useful.

But before choosing an optimization, measure the behavior you want to improve.

Without measurement, we risk solving the wrong problem, introducing unnecessary complexity, or optimizing a component that contributes very little to the overall performance of the system.

A technique that improves one workload may have little effect on another. Some optimizations also introduce trade-offs involving memory, consistency, operational complexity, and failure handling.

The goal is not to make every component faster in isolation.

The goal is to understand how the system behaves under a defined workload and improve the parts that matter most.

An assumption is not a benchmark

Suppose we believe an implementation is faster.

That is an assumption.

A benchmark gives us evidence.

Here, a benchmark means a repeatable measurement of a defined workload, not necessarily a full load, stress, or soak test.

A useful benchmark should define:

what is being measured;
the workload;
the environment;
the baseline;
the change;
the result.

It should also account for factors that can influence the result, such as warm-up time, concurrency, cache state, database state, network conditions, and background activity.

For example, comparing two API implementations using a handful of sequential requests may produce a different result from testing them under sustained concurrent traffic.

The first test might suggest that both implementations are fast. The second may reveal connection-pool exhaustion, increasing response times, or a growing queue of pending work.

The benchmark must reflect the question we are trying to answer.

If we want to know whether an implementation improves throughput, we should measure throughput under comparable conditions. If we want to know whether it improves the experience of individual users, we should pay close attention to latency.

A benchmark without a reliable baseline is difficult to interpret. A result without a repeatable process is difficult to trust.

Measure the right thing

For backend systems, performance is not just requests per second.

Useful measurements can include:

Latency: How long does a request or operation take?
Throughput: How much work does the system complete within a given period?
Error rate: How often do requests fail, time out, or return unexpected responses?
Memory usage: How much memory does the process consume, and does that usage continue to grow?
CPU usage: Is the application spending too much time performing computation?
Database time: How much time is spent executing queries and waiting for database operations?
Queue delay: How long does work wait before a worker begins processing it?
Resource saturation: Are CPU, memory, database connections, file descriptors, or other resources approaching their limits?

These measurements answer different questions.

A system can have high throughput while individual requests experience unacceptable delays. It can also have low average latency while a small but important percentage of requests take several seconds to complete.

That is why latency percentiles matter: they can expose slow requests that averages conceal, as described in the Google SRE Workbook’s monitoring guidance.

Consider these common metrics:

p50: The median request latency.
p95: The latency at or below which approximately 95% of requests in the measured sample complete.
p99: The latency at or below which approximately 99% of requests in the measured sample complete.

The p95 and p99 measurements can reveal slow requests that an average may conceal. They are particularly useful when a system has variable workloads, external dependencies, or occasional resource contention.

Error rate matters just as much. An implementation that processes more requests per second by rejecting or timing out more requests is not necessarily an improvement.

Performance measurements should therefore be interpreted together rather than in isolation.

Establish a baseline first

Before changing an implementation, we need to understand its current behavior.

A baseline provides a reference point for evaluating whether an optimization actually helps.

For an API endpoint, that baseline might include:

request latency at p50, p95, and p99;
throughput at a defined concurrency level;
error and timeout rates;
CPU and memory consumption;
database query duration;
database connection-pool utilization.

The exact measurements depend on the system and the performance question.

It is also important to record the conditions under which the measurements were collected. Runtime versions, hardware, container limits, database configuration, dataset size, and request patterns can all influence the outcome.

For example, an endpoint tested against a database containing a few hundred records may behave very differently when the production database contains millions.

A baseline does not need to be complicated. It needs to be representative, repeatable, and sufficiently detailed to support a meaningful comparison.

Measure the actual workload

A benchmark is only useful when its workload resembles the behavior we care about.

Consider an API that normally receives requests from many concurrent users. Testing it with one request at a time may hide contention and resource limitations.

A more representative test might account for:

the number of concurrent users or requests;
the proportion of reads to writes;
typical and unusually large payloads;
the size and distribution of database records;
cache hit and miss rates;
external service dependencies;
bursts of traffic and sustained load.

It is also useful to distinguish between different types of performance tests.

A load test evaluates behavior under an expected workload. A stress test pushes the system beyond its expected operating conditions to identify its limits and failure behavior. A soak test sustains a workload over a longer period to expose issues such as memory growth, resource leaks, or gradual degradation.

These tests answer different questions. A single benchmark rarely answers all of them.

We should also avoid treating a short-lived peak result as proof of sustainable performance. A system may perform well for a brief period before its connection pools, memory limits, or downstream dependencies become saturated.

Database performance

A slow API may not actually have a slow Node.js layer.

The database query may be the bottleneck.

That is why performance investigation needs to cross architectural boundaries.

HTTP
 │
 ▼
Application
 │
 ▼
Database

Every layer can contribute to latency.

The total response time may include network overhead, request parsing, application logic, database connection acquisition, query execution, result processing, serialization, and response transmission.

If we focus only on the application code, we may miss the operation consuming most of the time.

For example, an endpoint that retrieves a list of requisitions might execute a separate query for every requisition to retrieve its items. As the number of requisitions grows, the number of database queries grows with it.

This is a common form of the N+1 query problem: one query retrieves the parent records, then an additional query runs for each record to retrieve its related items.

Adding more Node.js workers will not necessarily solve it. In some cases, it may increase database contention by allowing more concurrent queries to reach an already overloaded database.

A better investigation would examine:

the number of queries executed per request;
query execution plans;
indexes and their effectiveness;
the volume of data retrieved;
unnecessary database round trips;
connection-pool utilization;
transaction duration and lock contention.

PostgreSQL's EXPLAIN ANALYZE executes the statement and reports actual runtime statistics alongside the query plan. Use care with statements that modify data, since their side effects occur.

The important point is that database performance should be measured at the database boundary, not inferred from API response time alone.

Understanding Node.js performance

When the database is not the bottleneck, we need to investigate the Node.js runtime and application code.

Node.js is well suited to I/O-heavy workloads, but that does not mean every Node.js application is automatically fast.

CPU-intensive JavaScript can block the event loop. As the Node.js event-loop guidance explains, long-running callbacks can reduce throughput and delay other clients. Large synchronous operations can delay unrelated requests. Excessive object creation can increase garbage-collection activity. Unbounded concurrency can overwhelm downstream services.

The appropriate response depends on the evidence.

For example:

If CPU usage is high and the event loop is frequently blocked, investigate expensive computations and synchronous operations.
If requests spend most of their time waiting for a database, investigate query execution, connection pools, and database capacity.
If memory usage grows continuously, investigate retained objects, unbounded collections, and resource cleanup.
If latency increases sharply under concurrency, examine resource contention, connection limits, queueing, and downstream saturation.

Node.js provides diagnostic capabilities such as CPU profiling and heap snapshots. The Node.js diagnostics guide describes tools that can help narrow down performance problems instead of relying on guesswork.

Profiling should generally be performed under a workload that reproduces the behavior being investigated. A CPU profile collected while the application is idle will not explain a CPU bottleneck that appears only under heavy traffic.

The objective is to connect an observed symptom to a specific cause.

Optimize the bottleneck, not the architecture diagram

Once we identify a bottleneck, we can select an appropriate optimization.

Caching may reduce repeated database reads. An index may improve query execution. A queue may move non-critical work out of the request-response path. Streams may reduce memory pressure when processing large amounts of data. Worker threads may help with CPU-intensive JavaScript.

But each technique solves a different problem.

Caching introduces invalidation and consistency concerns. Indexes consume storage and add overhead to writes, even as they can speed up retrieval (PostgreSQL documentation). Queues introduce delivery, retry, and idempotency considerations. Streams require correct backpressure handling. Worker threads introduce communication and lifecycle overhead.

Even clustering is not a universal solution. It can improve the use of multiple CPU cores for suitable workloads, but it does not automatically fix slow queries, inefficient algorithms, or overloaded external dependencies.

Before choosing an optimization, we should be able to explain:

What is the observed performance problem?
Which component or operation causes it?
Why should the proposed change improve that behavior?
What trade-offs could the change introduce?
Which measurements will demonstrate whether it worked?

These questions keep optimization tied to evidence rather than fashion.

Benchmarking belongs in engineering

As I continue studying advanced Node.js, I want benchmarking to become part of the engineering process rather than something performed only when a system becomes slow.

The sequence should be:

Measure
  ↓
Identify bottleneck
  ↓
Form a hypothesis
  ↓
Change
  ↓
Measure again
  ↓
Compare results

The comparison should use the same workload and sufficiently similar environmental conditions. Otherwise, we may attribute an improvement or regression to a code change when another variable actually caused it.

We should also retain the results of important performance investigations. A record of the baseline, hypothesis, change, and outcome helps the team understand why an optimization was introduced and whether it continues to provide value.

Not every optimization needs a dedicated benchmarking suite. Small, low-risk changes may only require focused measurements. Performance-critical paths, recurring workloads, and systems with strict latency or throughput requirements may justify automated benchmarks and regression checks.

The level of rigor should reflect the cost of getting the decision wrong.

Performance engineering is not about running benchmarks for their own sake. It is about making informed decisions and verifying their results.

Final thought

Optimization without measurement is speculation.

Production performance engineering requires evidence.

The fastest-looking implementation is not necessarily the fastest under realistic conditions. The most sophisticated architecture is not necessarily the most efficient. And a successful optimization is not one that merely looks cleaner or more advanced.

It is one that produces a measurable improvement without introducing unacceptable trade-offs.

Measure first. Understand the bottleneck. Make a targeted change. Measure again.

Going Deeper Into Node.js: Understanding the Runtime Behind the Framework

Tope Taiwo — Mon, 26 Oct 2026 06:35:15 +0000

"Go beyond framework APIs to understand the Node.js event loop, concurrency, streams, memory, networking, and graceful shutdown."

The more serious my backend engineering work becomes, the less satisfied I am with knowing only the framework APIs.

NestJS and Express are useful abstractions.

But eventually, understanding the runtime underneath them becomes important.

A framework can help organize an application, enforce architectural boundaries, and simplify common backend operations. It cannot eliminate the runtime behavior of the code we write.

When an application becomes slow, memory consumption increases, requests begin timing out, or background work interferes with API responsiveness, the explanation may have little to do with the framework itself.

It may be a consequence of Node.js executes JavaScript, manages asynchronous operations, handles data, or uses system resources.

That is why I consider runtime knowledge an important part of backend engineering.

The framework is not the runtime

When we write:

await service.execute();

there is a substantial amount happening underneath.

Node.js manages:

JavaScript execution.
Asynchronous operations.
Event-loop scheduling.
Networking.
Timers.
Streams.
Process behavior.

It also provides access to operating-system capabilities through its APIs and underlying runtime components.

Understanding these mechanisms helps explain application behavior.

For example, NestJS may handle dependency injection, route registration, guards, interceptors, and exception filters. Express may provide middleware composition and HTTP routing.

But neither framework changes the fundamental execution model of JavaScript on the main thread.

A poorly designed synchronous operation can still block request processing, regardless of how cleanly the application is organized.

Similarly, a memory-intensive operation can still exhaust the process's available memory, even when every service follows a well-defined architecture.

Framework abstractions simplify development. They do not remove the need to understand the system underneath them.

Why the event loop matters

Node.js is designed around asynchronous I/O.

That makes it effective for many backend workloads, particularly applications that spend substantial time waiting for databases, network responses, files, and other external resources.

But asynchronous syntax does not automatically make every operation cheap.

A CPU-heavy operation can still block JavaScript execution.

That means backend engineers need to understand the difference between:

I/O-bound work

and:

CPU-bound work

I/O-bound work spends much of its time waiting for an external operation to complete. CPU-bound work spends much of its time performing computations.

Consider a typical request:

const user = await userRepository.findById(userId);
const permissions = await permissionService.getForUser(user);
return permissions;

If the repository and permission service perform asynchronous I/O, the JavaScript execution can yield while those operations are pending.

This allows Node.js to make progress on other work rather than blocking the main thread while waiting for each response.

However, await does not move the underlying computation to another thread automatically.

When an awaited operation is CPU-intensive JavaScript, the computation still runs on the thread executing that JavaScript.

For example:

const result = expensiveCalculation(largeDataset);

If expensiveCalculation takes several seconds to execute synchronously, other JavaScript callbacks on that thread must wait.

This can increase API latency, delay timers, and prevent the application from responding promptly to other requests.

The important distinction is not whether a function is declared async.

It is what the function actually does and where that work executes.

Understanding the event loop beyond `async` and `await`

The event loop is central to how Node.js coordinates JavaScript execution and asynchronous operations.

At a high level, the runtime executes JavaScript callbacks and processes scheduled work as the relevant operations become ready.

Node.js uses the event loop and underlying mechanisms such as libuv to coordinate I/O, timers, and other asynchronous activity.

Promises and async/await also involve microtask scheduling, which affects the order in which callbacks execute.

For example:

console.log("A");

setTimeout(() => {
  console.log("B");
}, 0);

Promise.resolve().then(() => {
  console.log("C");
});

console.log("D");

In a normal execution of this example, the output is:

A
D
C
B

The synchronous statements execute first. The resolved promise's reaction runs as a microtask before the timer callback in this example.

The exact scheduling behavior becomes more nuanced when multiple event-loop phases, I/O callbacks, and different scheduling APIs are involved.

The practical lesson is that asynchronous code is not necessarily executed in the order in which developers intuitively expect it to be.

Understanding scheduling becomes particularly useful when debugging race conditions, unexpected callback ordering, delayed timers, and performance problems.

Asynchronous does not mean parallel

One distinction worth understanding is the difference between concurrency and parallelism.

Concurrency means multiple operations can make progress over overlapping periods.

Parallelism means multiple operations execute at the same time.

Node.js can handle many concurrent I/O operations without requiring a separate JavaScript thread for each request.

For example:

const [user, orders, settings] = await Promise.all([
  userRepository.findById(userId),
  orderRepository.findByUserId(userId),
  settingsService.getForUser(userId),
]);

If these operations are independent, Promise.all allows them to be initiated together rather than waiting for each one to finish before starting the next.

However, this does not mean the JavaScript computations inside all three operations execute in parallel on the main thread.

It also does not mean unlimited concurrency is desirable.

Launching thousands of database queries simultaneously can exhaust connection pools, overload the database, and increase latency across the application.

Concurrency must be managed according to the capacity of the dependencies involved.

For large workloads, bounded concurrency, batching, backpressure, and queue-based processing can be more appropriate than creating an unrestricted number of concurrent operations.

The goal is not simply to make more operations concurrent.

It is to make progress efficiently without overwhelming the system.

CPU-heavy work needs a different strategy

Some workloads are naturally expensive in terms of CPU.

Examples include:

Large data transformations.
Complex report generation.
Image processing.
Compression and decompression.
Cryptographic computations.
Computationally intensive parsing.

If such work executes synchronously on the main JavaScript thread, it can delay unrelated requests.

There are several possible responses, depending on the workload.

First, optimize the algorithm. Reducing unnecessary computation is often more effective than adding infrastructure.

Second, split work into manageable units where doing so is practical.

Third, use worker threads when CPU-intensive JavaScript can benefit from execution on another thread.

For example, Node.js provides the worker_threads module for running JavaScript in parallel across threads.

import { Worker } from "node:worker_threads";

const worker = new Worker(new URL("./calculation.worker.js", import.meta.url), {
  workerData: {
    input: largeDataset,
  },
});

This is an illustrative example of creating a worker; a production implementation also needs result handling, error handling, worker lifecycle management, and an appropriate strategy for reusing or pooling workers.

Creating a new worker for every small task can introduce unnecessary overhead.

For sustained workloads, a worker pool may be more suitable.

It is also important to distinguish worker threads from child processes. Worker threads execute JavaScript in separate threads within the same process, while child processes run in separate operating-system processes with their own memory spaces and lifecycle.

Neither approach automatically makes a workload efficient. The algorithm, data-transfer costs, available CPU resources, and operational requirements still matter.

Streams matter too

Large files and large data sets should not always be loaded completely into memory.

Streams provide a mechanism for processing data incrementally.

This becomes particularly relevant to:

Uploads.
Downloads.
Exports.
Imports.
Large transformations.

Consider an export that retrieves hundreds of thousands of database records.

One approach is to load every record into an array, transform the entire array, and then generate the output.

That can consume substantial memory, especially when several exports run concurrently.

A streaming approach processes data incrementally, allowing the application to work with smaller chunks rather than retaining the entire result in memory.

Conceptually:

Database Cursor
      │
      ▼
Transform Records
      │
      ▼
CSV Formatter
      │
      ▼
HTTP Response or File

Node.js streams support this kind of processing.

However, streams do not automatically make every pipeline memory-safe. The source must also avoid buffering the entire dataset, and the pipeline must handle backpressure correctly.

Backpressure allows a slower downstream consumer to influence how quickly upstream components produce data.

Without appropriate backpressure handling, a fast producer can overwhelm a slower consumer and accumulate data in memory.

For stream pipelines, Node.js provides utilities such as pipeline from node:stream/promises, which help coordinate stream completion and error propagation.

For large imports and exports, I also consider batching, database cursor behavior, cancellation, and what happens when the destination disconnects or fails.

The objective is not merely to use streams because the files are large.

It is to design a pipeline whose memory usage and failure behavior remain manageable as the workload grows.

Memory management and garbage collection

Memory behavior is another area where runtime knowledge becomes useful.

JavaScript provides automatic memory management, so developers do not ordinarily allocate and free memory manually.

However, automatic garbage collection does not mean memory consumption takes care of itself.

An application can retain references to objects it no longer needs. Large caches, unbounded collections, accumulated event listeners, and retained request data can all contribute to excessive memory usage.

For example:

const requestHistory: unknown[] = [];

app.use((req, _res, next) => {
  requestHistory.push({
    url: req.url,
    timestamp: Date.now(),
  });

  next();
});

This simplified example continually accumulates data without a retention policy.

In a long-running process, the collection can grow indefinitely.

The actual memory problem may not be a traditional leak in the sense of manually allocated memory that cannot be freed. It may be reachable data that the application continues to retain unnecessarily.

Garbage collection can reclaim objects that are no longer reachable, but it cannot reclaim objects that remain referenced.

Memory investigations should therefore consider:

Heap growth over time.
Retained objects and references.
Large buffers and external memory.
Cache size and eviction behavior.
Concurrent requests and temporary allocations.
Garbage-collection activity and its effect on latency.

It is also useful to distinguish the JavaScript heap from total process memory. Buffers and other native allocations can contribute to memory consumption outside the JavaScript heap.

This matters when an application approaches its container or operating-system memory limit even though the reported JavaScript heap does not appear unusually large.

Networking and connection management

Backend applications spend a substantial amount of time communicating with databases, caches, message brokers, and external APIs.

Understanding the runtime's networking behavior helps explain why connection management matters.

Consider a service that creates a new database connection for every request.

Even if each individual query is fast, the application may incur unnecessary connection setup overhead and overwhelm the database under concurrent load.

Connection pools help reuse connections and place limits on concurrency.

The same principle applies to Redis clients and HTTP clients, subject to each client's connection-management model.

Connection reuse alone, however, does not solve every performance problem.

A pool that is too small may create queues and increase latency. A pool that is too large may overload the dependency. Requests that wait indefinitely for a connection can consume resources without making progress.

Timeouts, cancellation, retry policies, and connection limits should reflect the behavior and capacity of the dependency.

The application should also distinguish between connection failures, timeouts, and application-level errors because they may require different recovery strategies.

Understanding networking helps developers reason about the complete request path rather than treating every delay as a slow function.

Process behavior and graceful shutdown

A Node.js application is also an operating-system process.

That process can receive termination signals, encounter uncaught failures, run out of memory, or be restarted by its process manager or container platform.

These events matter for HTTP servers and background workers alike.

A graceful shutdown strategy should allow the application to stop accepting new work, finish or safely interrupt in-flight operations where possible, close connections, and release resources.

For a background worker, this may also involve stopping the acquisition of new jobs and allowing active jobs to finish within a defined shutdown period.

The implementation depends on the framework, queue library, deployment platform, and required shutdown guarantees.

A process that exits cleanly during development may still lose in-flight work if production shutdown behavior has not been considered.

This is another reason runtime knowledge extends beyond JavaScript syntax.

It includes understanding how the application behaves throughout its lifecycle.

Framework knowledge is not enough

A senior Node.js engineer should understand what happens beneath the framework abstraction.

That is why I have been expanding my learning beyond controllers and services into:

Event-loop behavior.
Asynchronous execution.
Streams.
Networking.
Memory.
Worker threads.
Process behavior.
Performance measurement.

These topics are connected.

A slow endpoint may be waiting on a database, competing for a connection, performing synchronous CPU-heavy work, or buffering too much data.

A memory problem may originate in a cache, a stream pipeline, or the accumulation of objects retained by a long-running process.

A worker may be processing jobs correctly but still reduce API responsiveness if both workloads compete for the same CPU and memory resources.

Understanding the runtime helps narrow these possibilities before introducing unnecessary architectural complexity.

Measure before optimizing

Runtime knowledge becomes most useful when combined with measurement.

Performance problems should be investigated using evidence rather than assumptions.

Useful measurements include:

Request latency, including percentiles such as p95 and p99.
Throughput and error rates.
Event-loop delay and utilization.
CPU and memory consumption.
Garbage-collection behavior.
Database connection-pool saturation.
External dependency latency.
Stream throughput and buffering.
Background-job processing time and queue depth.

Node.js provides diagnostic and profiling capabilities, including performance hooks, CPU profiling, heap snapshots, and event-loop monitoring.

The right tool depends on the symptom being investigated.

For example, high event-loop delay may suggest that the JavaScript thread is spending too much time executing callbacks or performing synchronous work. It does not, by itself, identify the exact cause.

CPU profiles, heap snapshots, traces, and application metrics help establish what is actually happening.

The same principle applies to optimization.

An architectural change should address an observed bottleneck or a clearly defined reliability requirement.

Adding worker threads, Redis, additional processes, or more complex streaming infrastructure without understanding the problem can increase operational complexity without improving the system.

Final thought

Framework knowledge helps us build applications.

Runtime knowledge helps us understand why those applications behave the way they do.

That distinction becomes increasingly important as systems become more demanding.

As backend engineers, we should understand not only how to structure services and expose APIs, but also how the runtime executes work, manages resources, coordinates asynchronous operations, and responds to failure.

The framework is an important part of the application.

It is not a substitute for understanding the platform on which the application runs.

Testing Backend Architecture, Not Just Functions

Tope Taiwo — Fri, 23 Oct 2026 09:27:18 +0000

"Test backend boundaries, integration behavior, failure handling, and critical workflows to build confidence beyond unit-test coverage."

A backend can have excellent unit-test coverage and still be difficult to trust.

The reason is that functions are only one part of a system.

We also need to test boundaries.

A function may behave correctly in isolation while the repository it calls generates an incorrect query, the message publisher silently loses an event, or the HTTP layer returns an inappropriate response when an operation fails.

These are not necessarily unit-test failures. They are failures in the relationships between components.

When designing backend systems, I consider testing part of the architecture itself. The way application logic depends on infrastructure, the way data moves between layers, and the way failures propagate all influence how a system should be tested.

Different tests answer different questions

A unit test can ask:

Does this business rule behave correctly?

An integration test can ask:

Does this repository correctly communicate with PostgreSQL?

An API test can ask:

Does this endpoint correctly connect HTTP input to application behavior?

These are different questions.

Each test type provides a different kind of confidence.

Test type	Primary question
Unit tests	Does the isolated business logic behave correctly?
Integration tests	Do collaborating components work together correctly?
API tests	Does the HTTP contract behave as expected?
Contract tests	Do independently developed components agree on their interfaces?
End-to-end tests	Does a complete critical workflow work across the system?

These categories can overlap. An API test, for example, can be an integration test if it exercises the HTTP layer and application dependencies together.

The important distinction is not the label attached to a test.

It is the behavior being verified and the dependencies involved.

Test business logic independently

This is one reason I prefer application code to depend on interfaces.

A use case can be tested with fake implementations.

That means a test can focus on the business rule without requiring every infrastructure dependency.

Consider an application service that approves a requisition.

Its responsibilities might include checking whether the requisition is eligible for approval, verifying the actor's permissions through an appropriate application boundary, updating the requisition, and requesting any required follow-up work.

The business rule should not need a live Redis instance, an Azure Service Bus connection, or a running email provider merely to determine whether an approval is valid.

A simplified example might look like this:

interface RequisitionRepository {
  findById(id: string): Promise<Requisition | null>;
  save(requisition: Requisition): Promise<void>;
}

interface ApprovalPublisher {
  publish(event: RequisitionApproved): Promise<void>;
}

An application use case can depend on these interfaces rather than concrete infrastructure clients.

A unit test can then provide controlled implementations of the dependencies and verify the important behavior:

An eligible requisition can be approved.
An ineligible requisition is rejected.
An unauthorized actor cannot approve it.
The correct business state is produced.
The appropriate event or follow-up action is requested.

The test remains focused on the application's rules.

However, a fake repository cannot prove that the real repository correctly implements persistence. That is a separate responsibility requiring integration tests.

Abstraction makes isolated testing easier, but it does not remove the need to verify the real implementation.

Integration tests still matter

Abstraction does not eliminate integration testing.

If PostgreSQL is part of production, we need confidence that:

Queries work.
Constraints work.
Transactions work.
Mappings work.

The same applies to queues, caches and external integrations.

A repository can satisfy its TypeScript interface while generating an incorrect query. A database constraint may behave differently from an assumption encoded in a mock. A transaction may fail under conditions that a unit test never exercises.

For example, if a requisition approval must update the requisition and create an audit record in the same transaction, an integration test should verify the actual database behavior.

It should establish that both changes commit together and that a failure in either operation rolls back the relevant changes.

This is particularly important when using an ORM. The ORM's type safety does not guarantee that every query, migration, relation, or transaction behaves correctly in the deployed database.

Integration tests should use a real instance of the relevant dependency whenever its behavior is part of the contract being tested.

For PostgreSQL, this often means running tests against an isolated test database, commonly provisioned through a disposable container.

The goal is to verify the behavior of the real database rather than reproduce its behavior through a mock.

Test the boundaries between layers

A layered backend often separates controllers, application services, repositories, and infrastructure adapters.

Each boundary introduces a contract.

For example:

HTTP Controller
      │
      ▼
Application Use Case
      │
      ▼
Repository Interface
      │
      ▼
PostgreSQL Adapter

The controller translates HTTP input into an application request. The use case applies business rules. The repository adapter translates application operations into database operations.

Testing only the use case does not establish that the controller validates and maps input correctly.

Likewise, testing only the repository does not establish that the HTTP endpoint returns the correct status code or response structure.

The tests should reflect these responsibilities.

For an API endpoint, relevant assertions may include:

Invalid input produces the expected validation response.
Authentication and authorization are enforced.
The correct application use case is invoked.
Business errors map to appropriate HTTP responses.
Successful responses conform to the documented API contract.
Internal implementation details are not unintentionally exposed.

These checks help identify defects that are easy to miss when testing individual functions.

Test architecture through dependency direction

Testing also reveals whether the architecture has the boundaries it claims to have.

If a business use case can only be tested by starting PostgreSQL, Redis, a queue, and several external services, it is worth examining its dependencies.

That does not automatically mean the design is wrong. Some operations genuinely require integration testing.

However, if a simple business rule is inseparable from infrastructure initialization, the application may have unnecessary coupling.

A useful structure is:

Application Tests
       │
       ├── Fake Repository
       ├── Fake Message Publisher
       └── Fake External Service

Integration Tests
       │
       ├── Real PostgreSQL
       ├── Real Redis, where required
       └── Real Messaging Adapter, where required

The first group verifies application behavior without requiring the complete infrastructure.

The second group verifies that the infrastructure adapters honor their contracts.

This separation allows business rules to be tested quickly while retaining confidence in the actual integrations.

The objective is not to mock everything.

It is to control dependencies where appropriate and exercise real dependencies where their behavior matters.

Failure testing

A production-oriented test suite should also consider failure.

Examples:

Database unavailable
Queue publish fails
Email provider fails
Duplicate request arrives
Transaction rolls back
External API times out

The system's response to failure is part of its behavior.

A system should not be considered reliable simply because its successful execution path works.

Consider a background-job workflow. A job may be accepted successfully but fail during processing. The worker may retry it, encounter the same error repeatedly, or crash after performing an external side effect.

Tests should verify the application's expected behavior in these situations.

Depending on the workflow, that might mean:

A transient failure is retried according to policy.
A permanent failure is surfaced for investigation.
Duplicate processing does not repeat a protected business operation.
A database transaction rolls back when a required operation fails.
A failed external dependency does not leave the application in an invalid state.
Errors are recorded with enough context to support diagnosis.

These tests are especially valuable for asynchronous workflows because the producer, broker, and consumer do not share one execution context.

The test should verify the actual recovery behavior rather than merely assert that an exception was thrown.

Test transactional consistency

Database transactions and messaging introduce failure scenarios that ordinary happy-path tests may overlook.

Suppose an application creates an order and records an outbox event in the same database transaction.

A useful integration test verifies that both records are committed when the transaction succeeds and neither is committed when the transaction rolls back.

The publisher can then be tested separately to verify that pending events are delivered and their publication state is updated appropriately.

A further test should consider what happens when publication succeeds but the publisher fails before recording that success.

The event may be published again during recovery.

This is where idempotent consumers become important.

Testing the transactional outbox therefore involves more than checking that an event was inserted into a table. It includes persistence, publication, retry, and duplicate-delivery behavior.

The same principle applies to other business operations that coordinate multiple state changes.

Tests should verify the consistency guarantees the architecture is intended to provide.

Contract testing for replaceable infrastructure

When an application supports multiple infrastructure providers, contract testing becomes particularly useful.

For example, a messaging abstraction might have both a BullMQ adapter and an Azure Service Bus adapter.

Both are expected to satisfy the application's message-publishing contract.

Rather than testing each implementation against unrelated assumptions, define the shared behavior that every adapter must preserve.

This may include:

Correct message serialization.
Preservation of message identifiers and correlation metadata.
Correct handling of publication failures.
Appropriate error translation.
Compliance with the application's delivery contract.

The same reusable contract tests can then be run against each adapter where applicable.

However, shared contract tests should not pretend that provider-specific features are identical.

If one provider supports a particular scheduling or settlement behavior and another does not, that difference should be documented and tested explicitly.

Contract testing helps establish that implementations honor the abstraction. It does not prove that two providers have identical operational guarantees.

End-to-end tests should protect critical workflows

Unit and integration tests do not eliminate the need for end-to-end testing.

Some workflows cross enough boundaries that their combined behavior deserves verification.

Consider a workflow that starts with an authenticated HTTP request, writes a business record, creates an outbox event, publishes a message, and eventually updates the processing status.

An end-to-end test can verify that the complete workflow behaves correctly in a controlled environment.

However, end-to-end tests are usually more expensive and more sensitive to environmental failures than unit tests.

They should therefore focus on important business workflows rather than duplicate every assertion already covered by lower-level tests.

For example, a requisition approval system might prioritize end-to-end tests for:

Successful approval and audit recording.
Rejection of an unauthorized approval attempt.
Recovery from a failed asynchronous operation.
Prevention of duplicate business effects.
Correct handling of an invalid state transition.

The objective is to establish confidence in the complete workflow, not to make every test exercise the entire application.

Avoid testing implementation details

A test can pass consistently and still provide little confidence if it verifies the wrong thing.

For example, asserting that a service calls a particular private helper may couple the test to the current implementation rather than the intended behavior.

A harmless refactoring could then break the test without introducing a real defect.

Prefer assertions about observable outcomes and meaningful contracts.

For a business use case, that might mean verifying the resulting state, the emitted application message, or the error produced for an invalid operation.

For a repository, it might mean verifying persisted records and database constraints.

For an API, it might mean verifying the response status and payload.

Mocks remain useful, especially for controlling failure scenarios. The problem is not mocking itself; it is treating a mock's configured behavior as proof that the real dependency behaves the same way.

Tests should protect the system's intended behavior while allowing internal implementations to evolve.

Coverage is a signal, not the objective

Code coverage can help identify untested parts of a codebase.

It cannot, by itself, establish that the important business scenarios have been tested.

A test suite can execute every line of a function without verifying the correctness of its decisions.

Likewise, a system with a lower coverage percentage may have strong tests around its most critical business rules and failure boundaries.

Coverage is useful when interpreted alongside other evidence.

I would pay particular attention to:

Critical business rules and state transitions.
Authorization and access-control boundaries.
Transactional consistency.
Data integrity and database constraints.
Message delivery and duplicate-processing behavior.
Recovery from external dependency failures.
Important API contracts.

These areas often matter more to production confidence than achieving a particular percentage without considering what the tests actually establish.

Final thought

Testing should follow architectural boundaries.

The goal is not simply to achieve a percentage.

The goal is to establish confidence that the system behaves correctly at each important boundary.

Unit tests help verify business rules. Integration tests verify that real components work together. Contract tests protect agreements between independently implemented components. End-to-end tests establish confidence in critical workflows.

Each has a role.

A well-designed test suite should make it possible to change an infrastructure implementation, refactor application logic, or introduce a new integration without unnecessarily increasing the risk of regressions.

That is the purpose of testing backend architecture, not just functions.

Transactions, Events and the Problem of Reliable Side Effects

Tope Taiwo — Thu, 22 Oct 2026 08:32:21 +0000

"Understand the dual-write problem and how transactional outbox patterns, idempotent consumers, retries, and recovery improve event reliability."

A common backend operation looks simple:

Update database
      +
Publish event

The difficulty appears when one succeeds and the other fails.

This is a problem I consider when designing backend systems that coordinate database transactions with asynchronous processing, external services, or other independently operating components.

The challenge is not simply how to save a record or publish a message.

It is how to ensure that the system remains consistent when failures occur between those operations.

The dual-write problem

Suppose the application does:

1. Save order
2. Publish OrderCreated

If step 1 succeeds and step 2 fails, the database contains the order but downstream systems never receive the event.

The order may exist, but the systems responsible for inventory, notifications, reporting, or other follow-up operations may never learn that it was created.

Reverse the order and another failure mode appears.

1. Publish event
2. Save order

If the database operation fails after the event has been published, consumers may process an event for a transaction that never committed.

The problem exists because the application is trying to coordinate two independent operations without a shared atomic commit.

This is commonly known as the dual-write problem.

The two operations may each be reliable on their own. That does not make the combination reliable.

Transactions do not automatically solve messaging

A database transaction can provide atomicity within the database.

For example:

BEGIN
  │
  ├── Insert Order
  ├── Insert Order Items
  └── Update Inventory
  │
COMMIT

If an operation fails before the transaction commits, the database can roll back the changes covered by that transaction.

This is valuable because related business changes remain consistent.

However, a database transaction cannot automatically make an external message broker part of that same transaction.

Consider:

await prisma.$transaction(async (tx) => {
  const order = await tx.order.create({
    data: orderData,
  });

  await messageBus.publish({
    type: "OrderCreated",
    payload: {
      orderId: order.id,
    },
  });
});

At first glance, this appears to combine the database change and event publication into one operation.

It does not.

The message broker does not automatically participate in the PostgreSQL transaction.

If the message is published successfully but the database transaction later rolls back, consumers may process an event for an order that does not exist.

If the database commits but the application encounters a failure while publishing, the order may exist without the corresponding event.

Holding a database transaction open while calling an external service also increases transaction duration and can introduce unnecessary latency, resource contention, and failure dependencies.

The important distinction is that database atomicity does not imply atomicity across independent systems.

The transactional outbox pattern

This is where patterns such as the transactional outbox become useful.

The idea is straightforward: instead of attempting to publish an event directly as part of the database transaction, persist the intent to publish it in the same transaction as the business change.

Conceptually:

Database Transaction
      │
      ├── Business Data
      │
      └── Outbox Event
              │
              ▼
          COMMIT

Both are committed together.

A separate process can then publish the outbox event:

Outbox
  │
  ▼
Publisher
  │
  ▼
Message Bus
  │
  ▼
Consumers

If the transaction rolls back, neither the business change nor its outbox event is committed.

If the transaction commits, the outbox record remains available for publication, even if the messaging infrastructure is temporarily unavailable.

The publisher can retry delivery without requiring the original business operation to run again.

This changes the consistency problem.

Instead of trying to make a database write and a broker publication happen atomically, the application makes the business change and the intent to publish atomic. A separate process is responsible for delivering that intent.

The message may be published later, but the system has a durable record that publication is required.

A practical implementation

In a PostgreSQL-backed TypeScript application, the business record and outbox event can be persisted through the same Prisma transaction.

For example:

await prisma.$transaction(async (tx) => {
  const order = await tx.order.create({
    data: orderData,
  });

  await tx.outboxEvent.create({
    data: {
      eventId: crypto.randomUUID(),
      eventType: "OrderCreated",
      aggregateId: order.id,
      payload: {
        orderId: order.id,
      },
      status: "PENDING",
    },
  });
});

The example assumes an outbox model containing the corresponding fields.

The significant detail is that both operations use the same transaction client, tx.

If either operation fails, the transaction does not commit either change.

The outbox record might contain fields such as:

eventId: A stable identifier for the event.
eventType: The type of event to publish.
aggregateId: The identifier of the business entity associated with the event.
payload: The serialized event data.
status: The publication state.
createdAt: When the event was recorded.
publishedAt: When successful publication was recorded, if applicable.
attempts: The number of publication attempts.
lastError: Diagnostic information about the most recent failure.

The exact schema depends on the application's requirements.

The outbox is not simply a logging table. It represents durable work that remains to be delivered.

Its lifecycle, retention policy, indexing strategy, and recovery behavior therefore deserve the same attention as other operational components.

The publisher introduces another failure boundary

The transactional outbox solves the original dual-write problem, but it does not eliminate every failure mode.

Consider the publisher:

1. Read pending outbox event
2. Publish event
3. Mark event as published

What happens if the message broker accepts the event but the publisher crashes before step 3?

The outbox record still appears pending.

When the publisher retries, it may publish the same event again.

This means a transactional outbox generally supports reliable, retryable publication with possible duplicate delivery, not automatic exactly-once processing across the database and broker.

That distinction is important.

A robust implementation should assume that an event may be delivered more than once and design its consumers accordingly.

For example, a consumer can use the stable event identifier to detect duplicate processing. For database-backed business effects, a unique constraint on the processed-event identifier can help prevent the same logical event from being applied twice.

However, deduplication and external side effects require careful handling. Recording an event as processed and performing an external action are not automatically atomic either.

The outbox moves the reliability boundary. It does not remove the need to reason about it.

Consumer idempotency still matters

Suppose an OrderCreated event triggers an inventory reservation.

A consumer might receive the same event twice because the publisher retried it or the broker redelivered it.

Without appropriate safeguards, the inventory reservation could be applied twice.

A simplified processing design looks like this:

Receive Event
     │
     ▼
Check Event Identity
     │
     ├── Already Processed
     │       │
     │       ▼
     │   Skip Duplicate
     │
     └── Not Processed
             │
             ▼
       Apply Business Change
             │
             ▼
       Record Event Identity
             │
             ▼
          COMMIT

When the business change and processed-event record belong to the same database, they can often be committed in one transaction.

A uniqueness constraint on the event identifier provides an additional safeguard against concurrent duplicate deliveries.

If processing requires an external API call, however, the consumer must consider the possibility of a crash between the external operation and the local completion record.

Depending on the operation, a provider-supported idempotency key, a durable workflow, reconciliation, or another coordination mechanism may be necessary.

The correct approach depends on the business consequence of repeating the operation.

For example, generating a report twice may be tolerable. Applying the same financial adjustment twice may not be.

Idempotency is therefore not just a messaging concern. It is a property that should be considered when designing the business operation itself.

Polling versus change data capture

There is more than one way to publish outbox events.

Polling

A background publisher periodically queries the outbox table for pending events.

Outbox Table
     │
     ▼
Poll Pending Events
     │
     ▼
Publish to Broker
     │
     ▼
Update Publication State

Polling is relatively straightforward to implement and operate.

However, the implementation must account for concurrent publishers, batch sizes, event ordering, retry behavior, and the possibility of multiple workers attempting to claim the same record.

Database locking, appropriately designed claim operations, or other coordination mechanisms can help prevent unnecessary contention.

The polling interval also affects publication latency.

Change data capture

Another approach is to use change data capture (CDC) to observe committed database changes and feed outbox records into a messaging pipeline.

This can reduce the need for an application-managed polling loop and may be appropriate for systems with higher throughput or existing streaming infrastructure.

However, CDC introduces its own operational requirements, including connector management, replication-log retention, schema evolution, monitoring, and recovery.

Neither approach is universally better.

Polling can be a good starting point for a system with moderate throughput and straightforward operational requirements. CDC may be more suitable when the scale, latency requirements, or existing infrastructure justify the additional complexity.

The choice should follow the workload rather than the popularity of the pattern.

Event ordering and concurrent publishers

Reliable publication does not automatically guarantee that every consumer will observe events in the order the business produced them.

Suppose an order produces these events:

OrderCreated
OrderConfirmed
OrderDispatched

If multiple publishers process pending records concurrently, or the broker distributes messages across partitions or sessions, downstream processing may observe a different order unless ordering is explicitly supported and configured.

For many workflows, this is manageable because consumers validate the current business state or process events independently.

For workflows that require ordering, the design may need to preserve order per aggregate, use a stable partition or session key where supported, or prevent later events from overtaking earlier events for the same entity.

The requirement should be explicit.

Global ordering can constrain throughput unnecessarily. Per-aggregate ordering is often sufficient when events concern independent business entities.

The outbox schema and publisher strategy should reflect that distinction.

Retries, dead-lettering, and operational recovery

A publisher needs a clear response to failures.

Temporary broker unavailability may justify retries with exponential backoff. Invalid message contracts or permanently rejected messages may require investigation instead of endless retries.

A production design should establish:

Which failures are retryable.
How retry delays are calculated.
Whether retries have a maximum or continue until resolved.
How permanently failing events are isolated.
How operators inspect and replay failed events.
How long published records are retained.
How publication lag is monitored.

A dead-letter mechanism can isolate messages that cannot be processed automatically, but moving a message to a dead-letter queue does not resolve the underlying business problem.

Likewise, deleting a failed outbox record may remove the evidence that a required business event was never delivered.

Recovery should be deliberate and auditable.

Useful operational metrics include the number of pending events, the age of the oldest pending event, publication failures, retry counts, and the time between event creation and successful publication.

These signals reveal whether the asynchronous pipeline is keeping up with the business workload.

This connects to previous decisions

This is why database architecture and messaging architecture cannot always be designed independently.

The moment a business operation produces an external side effect, consistency becomes an architectural concern.

It also connects to how I approach application-level messaging boundaries.

An application should not need to know whether its event is eventually published through BullMQ, Azure Service Bus, or another messaging infrastructure. However, the application still needs a reliable way to record the obligation to publish that event.

The outbox provides that durable boundary between the database transaction and the messaging infrastructure.

The publisher can then implement the transport-specific behavior, while consumers handle idempotency and the business effects associated with the event.

These responsibilities are separate, but they must work together.

Not every operation needs an outbox. A best-effort cache refresh may not justify one. A business workflow that must not silently lose an event may.

The decision should reflect the consequences of failure.

Final thought

Distributed systems often fail at the boundaries between systems.

Understanding those boundaries is more valuable than memorizing patterns.

The outbox pattern is useful because it addresses one specific consistency problem: reliably connecting database state changes with asynchronous messaging.

It does not eliminate duplicate delivery, guarantee exactly-once execution, or make every external side effect atomic.

It gives the application a durable record of what must be published and a mechanism for recovering when the messaging infrastructure is unavailable.

That is the distinction that matters when designing reliable backend systems.

The goal is not to add patterns everywhere.

It is to understand which failure a pattern addresses, which guarantees it provides, and which responsibilities still belong elsewhere in the architecture.

Designing Background Jobs Without Infrastructure Coupling

Tope Taiwo — Wed, 21 Oct 2026 09:34:21 +0000

"Use messaging boundaries to keep application workflows independent of queue providers while preserving delivery guarantees, idempotency, and job semantics."

Background processing is an important part of backend systems.

Email delivery, notifications, reports, imports and other long-running operations do not always belong inside the HTTP request.

Some operations are too expensive to execute synchronously. Others depend on external services that may be slow or temporarily unavailable. In many cases, the API should accept a request, record the necessary business state, and allow a worker to complete the operation independently.

The architectural question is how to introduce asynchronous processing without coupling the entire application to one queue technology.

The direct approach

A service might do this:

await queue.add("send-email", payload);

That works.

For a small application, it may even be sufficient.

However, as the application grows, the queue client can start appearing throughout application services, controllers, and business workflows.

The problem is not the method call itself. The problem is the dependency it introduces.

If application services depend directly on BullMQ, for example, they become aware of concepts and behaviors associated with that infrastructure. Switching to Azure Service Bus may then require changes beyond the infrastructure layer.

The application may also become coupled to provider-specific options such as job priorities, delay settings, retry configuration, and scheduling APIs.

That is a dependency-management problem, not merely a queue-management problem.

Introduce a messaging boundary

A better boundary can be:

interface MessageBus {
  publish<T>(message: DomainMessage<T>): Promise<void>;
}

The application can then say:

await messageBus.publish({
  type: "EmailRequested",
  payload,
});

The infrastructure decides how to deliver it.

This follows the dependency inversion principle: application-level code depends on an abstraction, while concrete messaging infrastructure implements that abstraction.

The business operation expresses what needs to happen. The infrastructure determines how the message is transported.

A more complete TypeScript contract might look like this:

export interface MessageEnvelope<TPayload> {
  messageId: string;
  type: string;
  version: number;
  occurredAt: string;
  correlationId?: string;
  payload: TPayload;
}

export interface MessageBus {
  publish<TPayload>(message: MessageEnvelope<TPayload>): Promise<void>;
}

The envelope provides a stable place for message identity, schema versioning, timestamps, and request correlation.

These fields become useful when investigating failures across HTTP requests, background workers, and external services.

The version field also provides a foundation for evolving message contracts without assuming that every producer and consumer will be deployed simultaneously.

One important distinction: a message such as EmailRequested is generally an application or integration message, not necessarily a domain event. A domain event describes something meaningful that has happened in the business, such as OrderConfirmed. A command or request asks for an operation to be performed.

These concepts can share transport infrastructure, but they should retain their distinct meanings.

Separate publishing from processing

A messaging boundary is only one part of the architecture.

The consumer also needs a boundary between message delivery and business processing.

A simplified design looks like this:

HTTP Request
     │
     ▼
Application Service
     │
     ▼
MessageBus Interface
     │
     ▼
Messaging Adapter
     │
     ├── BullMQ / Redis
     │
     └── Azure Service Bus
                  │
                  ▼
              Consumer
                  │
                  ▼
          Application Handler
                  │
                  ▼
           Business Operation

The producer and consumer can now evolve independently of the underlying transport.

The HTTP application does not need to know how a worker receives messages. Likewise, the business handler does not need to know whether the message arrived through a Redis-backed queue or a cloud messaging service.

For example, a report-generation handler should be concerned with validating the request, generating the report, storing the output, and recording the result.

It should not need to know how BullMQ acknowledges a job or how Azure Service Bus settles a message.

Those concerns belong in the infrastructure adapter.

Why I care about this

I have specifically been designing systems where the messaging infrastructure should not be permanently tied to one provider.

For example:

MessageBus
   │
   ├── BullMQ / Redis
   │
   └── Azure Service Bus

The business operation should not have to change simply because the transport changes.

This is particularly useful when an application begins with a lightweight Redis-backed queue but later requires managed messaging infrastructure, different operational capabilities, or integration with other enterprise systems.

However, provider independence should be an architectural objective rather than a promise that switching providers will always be effortless.

The application should isolate provider-specific dependencies while explicitly defining the capabilities it requires.

A transport change may still require infrastructure configuration, deployment changes, operational adjustments, and changes to message routing.

The purpose of the boundary is to prevent those changes from unnecessarily spreading through the business logic.

Define the abstraction around application needs

An abstraction should describe the capabilities the application actually needs.

For example, if the application only needs to publish background work, a small interface may be sufficient:

export interface BackgroundJobPublisher {
  publish<TPayload>(message: MessageEnvelope<TPayload>): Promise<void>;
}

If the application needs delayed delivery, scheduling, or explicit routing, those capabilities should be introduced deliberately.

They should not be added merely because one provider exposes them.

Consider these examples:

// Application-level intent
await jobPublisher.publish(message);

Compare that with:

// Provider-specific behavior leaking into application code
await bullQueue.add("generate-report", payload, {
  attempts: 5,
  backoff: {
    type: "exponential",
    delay: 2_000,
  },
  removeOnComplete: true,
});

The second example is not inherently bad. Those options are useful when configuring BullMQ.

The architectural problem appears when business services must understand and reproduce those settings across multiple providers.

Instead, infrastructure configuration should translate application-level policies into provider-specific options.

For example, an application may define a retry policy for transient report-generation failures. The BullMQ adapter can translate that policy into its retry configuration, while an Azure Service Bus adapter can implement the appropriate delivery, redelivery, and dead-letter behavior supported by that platform.

The abstraction should remain small enough to be understandable and explicit enough to preserve the guarantees the application needs.

But abstraction must preserve semantics

A queue abstraction cannot pretend that every messaging platform behaves identically.

The design must account for the semantics the application actually needs.

This includes questions around:

Retries.
Ordering.
Acknowledgement and settlement.
Dead-lettering.
Idempotency.
Delivery guarantees.
Delayed delivery and scheduling.
Message expiration.
Concurrency and backpressure.

These differences matter because two providers may expose similar APIs while providing different operational guarantees.

For example, BullMQ manages jobs through Redis and provides job-oriented capabilities such as retries, delays, and concurrency controls.

Azure Service Bus provides managed messaging capabilities, including message settlement, delivery-count handling, and dead-letter queues.

A successful method call to publish a message also does not mean the business operation has completed. It means the message was accepted according to the publishing mechanism's guarantees.

The consumer may execute later, fail, retry, or encounter a permanent error.

The application must define what success means at each stage.

Delivery guarantees and idempotency

Distributed messaging systems can deliver the same logical work more than once.

A worker may complete an external operation and then fail before acknowledging the message. The infrastructure may redeliver the message because it cannot establish that processing completed successfully.

This means an application should not assume that a job handler executes exactly once merely because its queue has retries or acknowledgement support.

Consider a notification workflow:

Message Received
       │
       ▼
Send Notification
       │
       ▼
Record Completion
       │
       ▼
Acknowledge Message

If the process crashes after sending the notification but before recording completion, a retry could send the notification again.

An idempotent consumer is designed to handle repeated delivery without repeating the business effect unnecessarily.

One approach is to assign a stable message identifier and record processed messages in a durable store with a suitable uniqueness constraint.

However, a deduplication record alone does not make every external side effect atomic. If sending an email and recording its completion are separate operations, a crash can still occur between them.

For external effects, the design may need provider-supported idempotency keys, durable delivery records, reconciliation, or a workflow that explicitly handles uncertain outcomes.

The correct solution depends on the business operation.

For example, duplicate report-generation work may be acceptable if the output is safely replaceable. Duplicate financial transfers are not.

The messaging abstraction must not hide these differences.

The database transaction problem

One of the most important failure cases appears when an application writes to a database and publishes a message as part of the same business operation.

Consider this sequence:

1. Save Order to PostgreSQL
2. Publish OrderConfirmed

What happens if the database transaction commits successfully but the message publication fails?

The order exists, but downstream consumers may never receive the event.

Reversing the order does not eliminate the problem. The message might be published successfully while the database transaction subsequently rolls back.

This is the dual-write problem.

For business operations that require reliable event publication, the transactional outbox pattern is a common solution.

Database Transaction
        │
        ├── Save Business Record
        │
        └── Save Outbox Message
                  │
                  ▼
             Transaction Commit
                  │
                  ▼
            Outbox Publisher
                  │
                  ▼
              Message Bus
                  │
                  ▼
               Consumer

The business record and its outbox message are committed in the same database transaction.

A separate publisher reads pending outbox records and publishes them to the configured messaging infrastructure. After successful publication, it records the appropriate delivery state.

The publisher may still crash after sending a message but before marking the outbox record as published. Therefore, duplicate publication remains possible, and consumers should be designed accordingly.

The outbox pattern does not create a distributed transaction between PostgreSQL and the queue. It makes the intent to publish durable and recoverable.

It is particularly useful for order processing, approvals, financial workflows, and other operations where losing an event would create an inconsistent business state.

For a best-effort cache refresh or noncritical notification, the added complexity may not be justified. The reliability requirement should determine the pattern.

Publishing is not the same as processing

A clean messaging boundary should distinguish the acceptance of a message from the completion of its work.

For example:

await messageBus.publish(message);

This should not imply that the email has been delivered or the report has been generated.

It should mean that the publishing operation completed according to the contract defined by the adapter.

The application may return a response such as:

{
  "status": "accepted",
  "messageId": "msg_123"
}

The exact response depends on the API contract.

If the client needs to know when processing finishes, the application needs an additional mechanism, such as a persisted job-status resource, a notification, or a polling endpoint.

This distinction prevents an asynchronous API from promising synchronous completion.

It also helps separate three different concerns:

Acceptance: Was the work successfully submitted?
Execution: Did a worker process it successfully?
Business completion: Was the intended business outcome achieved and recorded?

Those states may occur at different times and should not be conflated.

Avoid building a generic messaging framework

There is a temptation to create an abstraction that supports every conceivable queue feature.

For example:

interface UniversalQueue {
  publish(): Promise<void>;
  schedule(): Promise<void>;
  cancel(): Promise<void>;
  retry(): Promise<void>;
  acknowledge(): Promise<void>;
  deadLetter(): Promise<void>;
  changePriority(): Promise<void>;
}

This can become a lowest-common-denominator abstraction that either exposes provider-specific details everywhere or fails to express the capabilities of individual providers accurately.

Not every messaging technology supports the same operations in the same way.

A better approach is to define a small core contract and introduce additional interfaces only when the application genuinely requires those capabilities.

For example:

interface MessagePublisher {
  publish<TPayload>(message: MessageEnvelope<TPayload>): Promise<void>;
}

interface ScheduledMessagePublisher extends MessagePublisher {
  publishAt<TPayload>(
    message: MessageEnvelope<TPayload>,
    scheduledAt: Date,
  ): Promise<void>;
}

Even this extension should be introduced only when scheduling is a real application requirement.

If scheduling semantics differ between providers, the contract must define what scheduledAt means, what happens when a message is late, and how cancellation or rescheduling works.

A smaller abstraction with explicit limitations is often more maintainable than a universal interface that promises capabilities it cannot consistently deliver.

Organize the TypeScript implementation around boundaries

In a TypeScript backend, one possible feature-oriented structure is:

src/
├── modules/
│   └── notifications/
│       ├── application/
│       │   ├── ports/
│       │   │   └── message-publisher.ts
│       │   └── use-cases/
│       │       └── request-notification.ts
│       ├── domain/
│       │   └── notification.ts
│       └── infrastructure/
│           ├── messaging/
│           │   ├── bullmq-message-publisher.ts
│           │   └── service-bus-message-publisher.ts
│           └── consumers/
│               └── notification-consumer.ts
├── shared/
│   └── messaging/
│       ├── message-envelope.ts
│       └── message-registry.ts
└── bootstrap/
    └── messaging.ts

This is an illustrative structure rather than a mandatory folder convention.

The application layer owns the port it needs. Infrastructure adapters implement that port. The bootstrap layer selects and configures the implementation.

For example, dependency injection can bind the application interface to a BullMQ-backed implementation in one deployment and an Azure Service Bus-backed implementation in another.

The consumer's business handler can remain independent of the transport, while a provider-specific consumer adapter handles delivery, settlement, retries, and error translation.

Message schemas should also be validated at the system boundary. TypeScript types disappear at runtime, and a message may have been produced by an older application version or another service.

A typed interface alone is not sufficient to guarantee that a received message has a valid payload.

Testing the boundary

A messaging abstraction also makes application services easier to test.

A unit test can replace the real message publisher with a fake implementation that records published messages.

The test can then verify that the application requests the expected operation without requiring Redis, Azure credentials, or a running worker.

Separate integration tests should verify that each adapter correctly maps application contracts to provider behavior.

For example, adapter tests can cover:

Successful publication.
Transient infrastructure failures.
Invalid or incompatible messages.
Retry and dead-letter behavior.
Duplicate delivery.
Serialization and schema validation.
Connection loss and recovery.

The goal is to test both the application-level intent and the infrastructure-level guarantees.

A passing unit test does not prove that a queue configuration is reliable in production. Likewise, an integration test against one provider does not establish that another provider has identical semantics.

Final thought

The goal is not to create a generic "queue wrapper."

The goal is to create a meaningful application-level messaging boundary while keeping infrastructure replaceable.

That boundary should protect business logic from provider-specific APIs without hiding the operational realities of asynchronous processing.

Retries still require careful policies. Consumers still need idempotency. Critical database changes may require an outbox. Delivery guarantees must be understood rather than assumed.

A well-designed abstraction does not make these problems disappear.

It gives the application a clear place to define its requirements and gives the infrastructure a clear responsibility for implementing them.

That is the difference between simply hiding a queue client behind an interface and designing a backend that can evolve its messaging infrastructure without unnecessarily rewriting its business logic.

Redis Is More Than a Cache

Tope Taiwo — Tue, 20 Oct 2026 09:07:54 +0000

Explore Redis for caching, rate limiting, sessions, coordination, messaging, and background jobs, with attention to trade-offs and failure modes

Redis is frequently introduced to developers as a caching system.

That is only one of its uses.

In backend architecture, Redis can participate in several different patterns, including caching, session management, rate limiting, distributed coordination, real-time messaging, and background-job infrastructure.

Its usefulness comes from more than simply storing data in memory. Redis provides data structures and operations that can help applications solve specific problems without repeatedly querying a database or performing expensive computations.

However, using Redis effectively requires understanding what each pattern is responsible for, what guarantees it provides, and what happens when things fail.

Cache and queue are different responsibilities

A cache answers:

Can I retrieve this expensive result without recomputing it?

A queue answers:

Can I defer this piece of work and process it asynchronously?

They are different problems.

A technology can support both without making the concepts identical.

For example, an application might use Redis to cache a user's profile while also using BullMQ to schedule email delivery, report generation, or other background tasks.

The cached profile is optimized for repeated reads. The queued job represents work that must be executed, retried, or scheduled according to the queue's processing rules.

These responsibilities should remain separate in the application architecture, even when they share the same Redis infrastructure.

Read-through caching

A useful cache pattern is:

Request
  │
  ▼
Cache
  │
  ├── Hit → Return
  │
  └── Miss
        │
        ▼
     Database
        │
        ▼
      Cache
        │
        ▼
      Return

In a read-through-style flow, the application retrieves data from the cache first. When the requested data is missing, it reads from the database and populates the cache.

Strictly speaking, this is often implemented as a cache-aside pattern: the application explicitly manages cache reads, database reads, and cache population.

For example, a NestJS service might use Redis to cache frequently accessed organizational settings, product catalog entries, or permission metadata.

async function getOrganizationSettings(
  organizationId: string,
): Promise<OrganizationSettings> {
  const key = `organization:${organizationId}:settings`;

  const cached = await redis.get(key);

  if (cached !== null) {
    return JSON.parse(cached) as OrganizationSettings;
  }

  const settings = await organizationRepository.getSettings(organizationId);

  await redis.set(key, JSON.stringify(settings), {
    EX: 300,
  });

  return settings;
}

This example uses a five-minute expiration.

The actual implementation should also validate cached data, handle serialization failures appropriately, and decide what happens when Redis is unavailable.

The cache should improve the application's performance without becoming an accidental prerequisite for retrieving data that remains available in the primary database.

Cache invalidation is the difficult part

Adding a cache is easy.

Keeping cached data correct is harder.

Questions include:

When does the cache expire?
When is it invalidated?
What happens after an update?
What happens if the database succeeds but cache invalidation fails?
Can stale data be tolerated?

These are architectural questions.

Consider an organization whose approval limit changes from ₦500,000 to ₦1,000,000.

If the database is updated but Redis still contains the previous limit, the application may continue making decisions based on outdated information.

For display-oriented data, a short period of staleness may be acceptable. For authorization, financial controls, or approval decisions, it may not be.

Common strategies include:

TTL-based expiration: Allow entries to expire after a defined period.
Explicit invalidation: Delete or update affected cache entries after a successful database write.
Versioned cache keys: Include a version in the key to prevent old entries from being reused.
Write-through approaches: Update the database and cache through a coordinated application strategy.
Event-driven invalidation: Publish changes and invalidate related entries through a reliable event-processing mechanism.

None of these strategies eliminates every failure mode.

For example, if a database transaction commits successfully but the application crashes before invalidating Redis, the cache can remain stale.

For important workflows, an outbox pattern can help: record the business change and an event in the same database transaction, then process that event asynchronously to invalidate or refresh the cache.

The key lesson is that cache consistency must be designed, not assumed.

Redis data structures are part of the value

Redis is not limited to storing serialized JSON strings.

It supports data structures with operations suited to different access patterns.

Data structure	Useful applications
Strings	Cache entries, counters, temporary tokens
Hashes	Compact representations of objects and fields
Sets	Unique membership, feature assignments, membership checks
Sorted sets	Leaderboards, priority ordering, scheduled work indexes
Lists	Simple ordered collections and queue-like operations
Streams	Append-only event records and consumer groups
Bitmaps	Compact tracking of binary states
HyperLogLogs	Approximate distinct-count analytics

The important distinction is between choosing Redis because it is available and choosing a Redis data structure because its operations match the problem.

For example, a sorted set can support a leaderboard without repeatedly sorting an entire dataset in application memory.

A set can efficiently track membership, such as which feature flags have been assigned to a particular group.

A counter can support lightweight operational metrics, subject to appropriate expiration, cardinality, and retention policies.

The data structure should follow the access pattern and consistency requirements.

Rate limiting

Rate limiting is another common use of Redis.

An API may need to restrict how frequently a client can make requests within a given period.

For example:

Incoming Request
      │
      ▼
Identify Client
      │
      ▼
Check Rate Limit in Redis
      │
      ├── Within Limit → Continue
      │
      └── Limit Exceeded → Reject

A Redis-backed limiter can maintain counters shared across multiple application instances.

Without shared state, an application running on four replicas could enforce four independent limits instead of one limit across the entire deployment.

Redis operations such as INCR and expiration can implement simple fixed-window limits. However, the counter update and expiration setup must be designed to avoid race conditions, and a fixed window can allow bursts around window boundaries.

Depending on the requirements, a sliding-window or token-bucket algorithm may be more appropriate.

Rate limiting is useful for:

Login and password-reset endpoints.
Public APIs.
OTP and verification-code requests.
Expensive report-generation endpoints.
Protection against accidental request bursts.

The key design questions include the limiting identity, the time window, the permitted burst size, and the behavior when Redis becomes unavailable.

A global rate limit, a per-user limit, and a per-IP limit are different policies. They should not be treated as interchangeable.

Session management and temporary state

Redis can also store temporary application state.

Examples include:

Server-side sessions.
Short-lived verification challenges.
Temporary workflow state.
One-time application tokens.
Expiring reservations or locks.

This is particularly useful when an application runs across multiple instances.

If a session exists only in the memory of one Node.js process, a subsequent request routed to another instance may not find it.

Shared session storage allows instances to retrieve the same session state.

However, not every token should automatically be stored in Redis. The application must define expiration, revocation, access controls, and the consequences of Redis data loss.

For example, a password-reset challenge should expire and should not remain valid indefinitely simply because its key was never explicitly deleted.

Temporary state is still security-relevant state.

Distributed locks and coordination

Suppose several application instances attempt to generate the same monthly report simultaneously.

Or imagine multiple workers trying to perform the same scheduled reconciliation.

Redis can help coordinate such work through carefully designed locking or deduplication mechanisms.

A simplified lock acquisition might look like:

SET lock:monthly-report token NX PX 30000

Here:

NX requests creation only if the key does not already exist.
PX 30000 gives the lock a 30-second expiration.
token should be a unique ownership value.

The owner must release the lock only if the stored token still matches its own token. Otherwise, an expired lock could be acquired by another worker, and the original worker could accidentally delete the new owner's lock.

A simple lock implementation is not automatically a complete distributed-locking solution.

Long-running work, process pauses, network partitions, lock expiry, and delayed operations can all create concurrency problems.

For operations where duplicate execution could cause serious damage, consider database constraints, idempotency keys, transactional guarantees, or fencing tokens in addition to coordination.

A lock can reduce contention, but it should not replace the underlying correctness guarantees of the business operation.

Background jobs with BullMQ

Redis can also support queue infrastructure such as BullMQ.

That creates a useful separation:

Application
    │
    ├── Cache
    │
    └── Job Queue
           │
           ▼
        Workers
           │
           ├── Email delivery
           ├── Report generation
           ├── File processing
           └── Notifications

But the application should not confuse those responsibilities.

Imagine an API endpoint that accepts a request to generate a large report.

Generating the report inside the HTTP request can increase response times and consume application resources while the client waits.

Instead, the API can validate the request, persist the necessary business state, enqueue a job, and return a job identifier.

A worker processes the job independently.

BullMQ supports capabilities such as delayed jobs, retries, repeatable scheduling, concurrency controls, and job prioritization.

These capabilities are useful, but they do not mean every queued job will execute exactly once.

A worker may complete an external side effect and crash before acknowledging the job. The queue may then retry it.

Consequently, a production job handler should be designed to tolerate repeated execution.

For example, an invoice-notification job should check whether the relevant notification has already been recorded or use an appropriate idempotency mechanism before performing a non-repeatable side effect.

Similarly, retry policies should distinguish transient failures from permanent failures. Repeatedly retrying an invalid request does not make it valid.

The queue is responsible for managing work. The application is responsible for making that work safe to execute.

Pub/Sub and Redis Streams

Redis can also support communication between application components.

However, Pub/Sub and Streams serve different purposes.

Redis Pub/Sub allows publishers to send messages to subscribers. It is useful for transient notifications, but messages are not retained for subscribers that are disconnected when the message is published.

For example, an application could publish a notification that a dashboard should refresh its data.

Redis Streams provide a retained, ordered stream of entries, with consumer-group capabilities for distributing processing across consumers and tracking pending messages.

Streams can be useful for event-processing workflows that require consumers to resume work after interruptions.

Even then, retention, acknowledgment, pending-message recovery, consumer coordination, and duplicate handling must be designed explicitly.

Neither Pub/Sub nor Streams should automatically be treated as a substitute for a durable business record in a relational database.

The appropriate choice depends on whether the application needs transient notification, retained event processing, or a durable source of business truth.

Redis is not automatically the source of truth

One of the most important architectural decisions is determining which system owns the authoritative version of a piece of data.

For many business applications, PostgreSQL or another primary database remains responsible for durable business records, constraints, and transactions.

Redis can then support performance, coordination, temporary state, and asynchronous processing.

For example:

PostgreSQL
    │
    ├── Authoritative business records
    ├── Transactional constraints
    └── Durable workflow state

Redis
    │
    ├── Cached reads
    ├── Rate-limit counters
    ├── Temporary state
    └── Queue and coordination infrastructure

This is an architectural choice, not a universal rule that Redis can never be a primary datastore.

Redis supports persistence and other durability-related features. However, its actual recovery guarantees depend on the configuration, persistence mode, replication, failover behavior, and operational setup.

If losing a piece of data would violate a business requirement, its storage and recovery strategy must be evaluated accordingly.

A cache entry can usually be rebuilt.

An unrecorded payment, an unprocessed business obligation, or a lost approval decision may have much more serious consequences.

Those data categories should not receive the same reliability assumptions.

Redis introduces operational trade-offs

Redis can improve application performance, but it also introduces infrastructure and operational costs.

Memory management

Redis is commonly used as an in-memory datastore. Memory is therefore a resource that must be monitored and managed.

Applications should consider:

Memory limits and eviction policies.
Key expiration and retention.
Large values and expensive data structures.
Unexpected growth in the number of keys.
Cache churn and eviction rates.

An eviction policy that works well for a disposable cache may be inappropriate for data that the application expects to retain.

Connection management

Creating a new Redis connection for every request is usually a poor approach.

Node.js applications should generally reuse managed connections, configure sensible timeouts, and handle reconnection behavior deliberately.

Connection pooling or multiplexing should follow the behavior of the selected Redis client and workload.

Redis outages

A cache outage and a queue outage have different consequences.

If Redis is used only as a cache, the application might temporarily fall back to the database, provided that doing so will not overwhelm it.

If Redis also backs background-job infrastructure, an outage may prevent jobs from being enqueued or processed.

A fallback strategy must account for these differences.

For example, sending every cache miss directly to PostgreSQL during a Redis outage can create a database traffic spike. A controlled fallback, load shedding, or temporary rejection of nonessential requests may be safer.

Likewise, silently discarding a job because Redis is unavailable may violate the business contract of the endpoint.

Resilience begins with identifying which operations can fail open, which must fail closed, and which require durable recovery.

Monitoring

A production deployment should monitor Redis as an infrastructure dependency.

Useful signals include:

Memory consumption and evictions.
Command latency and slow operations.
Connection counts and rejected connections.
Cache hit and miss rates.
Queue depth and oldest-job age.
Failed, stalled, and repeatedly retried jobs.
Replication health and persistence-related failures where applicable.

These metrics help distinguish an application problem from an infrastructure problem.

A cache hit rate alone is not sufficient evidence that Redis is improving the system. The cache must also reduce meaningful latency or database load without introducing unacceptable staleness or operational complexity.

A practical rule for backend engineers

Before introducing Redis into a feature, answer these questions:

What problem are we solving? Is it repeated computation, request throttling, coordination, temporary state, or deferred work?
What is the source of truth? Can the data be reconstructed, or must it survive failure?
What consistency guarantees are required? Can the application tolerate stale data or duplicate processing?
What happens when Redis is unavailable? Can the operation safely fall back, wait, retry, or fail?
How will the feature be monitored? Which metrics will show that the design is working?
Is Redis necessary? Would a database constraint, an existing queue, or a simpler application-level solution solve the problem with fewer moving parts?

The goal is not to use Redis in as many places as possible.

It is to use it where its data structures and operational characteristics provide a clear advantage.

Final thought

Redis is a useful infrastructure component.

The important engineering skill is not knowing that Redis exists.

It is understanding which problem Redis is solving in a particular part of the architecture.

A cache should have an explicit consistency strategy. A queue should have a retry and idempotency strategy. A rate limiter should have a defined policy. A distributed lock should have clear ownership and failure semantics.

Those decisions are what separate simply adding Redis to a backend from designing a backend that uses Redis responsibly.

Why Choose RESTApi for Notifications?

Tope Taiwo — Mon, 19 Oct 2026 05:00:00 +0000

Choose REST, polling, SSE, or WebSockets for notifications based on delivery needs, while keeping notification processing reliable and asynchronous.

Not every notification system needs WebSockets.

That is one of the architectural decisions I have been thinking through while designing backend communication systems.

Real-time communication has its place, but introducing it by default can add complexity without necessarily improving the user experience. The better approach is to understand the requirements first and select the communication model that satisfies them.

Start with the requirement

Before selecting WebSockets, Server-Sent Events (SSE), or polling, we should ask:

What does the client actually need?

If the requirement is to:

Retrieve notifications.
Mark notifications as read or unread.
Manage notification preferences.
Retrieve notification data when the client requests it.
View notifications when opening a page or refreshing a list.

REST can be sufficient.

An HTTP-based API provides a straightforward contract for retrieving notifications, updating their read status, and managing preferences. It also fits naturally into applications that already use REST for their other backend interactions.

The important point is that retrieving a notification and receiving it immediately are different requirements.

If users only need to see their notifications when they open a page, refresh a list, or make a scheduled request, persistent connections may not be necessary.

However, if the application must deliver new notifications to an already-open client with minimal delay, polling alone may not be enough. That requirement should influence the design.

Why avoid unnecessary real-time complexity?

Real-time communication introduces additional concerns:

Persistent connections and their lifecycle.
Reconnection and recovery after network interruptions.
Connection management across multiple application instances.
Load balancing and connection distribution.
Connection state and resource consumption.
Authentication and authorization throughout the connection lifecycle.
Observability for connection failures and delivery delays.

These problems can be justified when the product requires real-time updates.

But they should be driven by requirements, not by the assumption that a more sophisticated communication mechanism is automatically a better one.

For example, WebSockets may be appropriate for collaborative applications, live operational dashboards, chat systems, or workflows where clients need frequent bidirectional communication.

For a conventional notification centre, however, an HTTP endpoint may provide everything the client needs.

A simpler communication model can reduce operational overhead, make the API easier to test, and allow the team to focus on notification correctness rather than connection management.

That does not mean REST is always cheaper or better. It means the additional complexity of real-time infrastructure should deliver a measurable benefit.

Notifications can be asynchronous internally

Choosing REST for the client-facing API does not mean notification generation has to be synchronous.

This distinction is important.

A business operation should not necessarily have to wait for an email, SMS, push notification, or in-app notification to be generated before it can complete.

The architecture can be:

Business Event
      │
      ▼
Message / Job
      │
      ▼
Notification Worker
      │
      ▼
Notification Store
      │
      ▼
REST API
      │
      ▼
Client

For example, when a requisition is approved, the business operation can record the approval and initiate the notification workflow. A background worker can then process the notification and persist the resulting in-app notification.

The client retrieves that notification through REST.

This separates three concerns:

Business processing: The operation that causes the notification.
Notification processing: The creation and handling of the notification, potentially through a background job.
Notification retrieval: The API through which the client accesses the notification.

Each concern can evolve independently, provided the boundaries and contracts remain clear.

There is also an important reliability consideration: if the business transaction commits but the notification job is never published, the user may not receive the expected notification. For workflows where this matters, a transactional outbox can help ensure that the intent to publish a notification is recorded atomically with the business change.

Background processing also requires appropriate retry policies, idempotency, and failure monitoring. A job should not create duplicate notifications every time a worker retries an operation.

REST simplifies the client-facing communication boundary; it does not remove the need for reliable asynchronous processing.

Preferences belong in the design

A notification system also needs preferences.

Different users or identities may have different preferences for:

Email.
SMS.
Push notifications.
In-app notifications.

Those decisions should be represented explicitly rather than hard-coded into business workflows.

For example, a business service should not need to know whether a particular user wants email notifications or push notifications. It should communicate the relevant business event or notification intent to the notification system, which can apply the appropriate preferences and delivery rules.

Preferences may also depend on notification category, urgency, organization policy, or whether a channel is available and supported.

However, preferences should not be confused with mandatory operational messages. Security alerts, compliance notices, and other critical communications may require different rules from ordinary informational notifications.

Separating these concerns makes the system easier to maintain and allows notification channels to evolve without forcing changes throughout the business domain.

What if real-time delivery becomes necessary?

Choosing REST today does not mean ruling out real-time delivery permanently.

The system can begin with a REST API for notification retrieval and introduce polling, SSE, or WebSockets later if actual product requirements justify the additional complexity.

Each option has different characteristics:

REST with polling: The client periodically requests updates. It is straightforward, but frequent polling can generate unnecessary requests.
Server-Sent Events (SSE): The server streams updates to the client over a persistent, one-way connection. It can suit notification feeds where the server needs to push updates and the client does not need bidirectional messaging over the same connection.
WebSockets: The client and server maintain a bidirectional communication channel. This is useful when both sides need to exchange frequent real-time messages.

Regardless of the delivery mechanism, the persisted notification record should remain the source of truth. A real-time event can tell the client that new data is available, but the client should not have to rely on a transient connection to recover notifications it missed while offline.

That separation also makes it easier to support multiple client types, including web and mobile applications, without tying notification history to a particular connection.

Final thought

Technology selection should follow requirements.

If real-time delivery is necessary, real-time infrastructure may be justified.

If it is not, REST can provide a simpler and more maintainable communication boundary.

The goal is not to choose the technology with the most capabilities. It is to choose the simplest architecture that satisfies the requirements reliably, while leaving room for the system to evolve when those requirements change.

What a Failed Import Taught Me About Long-Running Data Operations

Tope Taiwo — Sun, 18 Oct 2026 05:00:00 +0000

Learn how batching, durable progress tracking, idempotency, and targeted retries make long-running data imports recoverable after failures.

When seeding initial data into a database, large data imports can be deceptively simple.

The first implementation often looks like:

Read records
   ↓
Insert records
   ↓
Done

That model changes when an operation involves thousands of records and a network failure occurs.

The challenge is no longer just inserting data. It is ensuring that the operation can recover safely when something interrupts its progress.

The problem with treating an import as one operation

If an import processes thousands of records inside one enormous transaction, a failure can make recovery difficult. The transaction may hold locks for too long, consume substantial database resources, or exceed configured timeouts.

If it processes records individually, performance may suffer because of repeated database round trips and transaction overhead.

A practical approach is batching.

Records
 │
 ├── Batch 1
 ├── Batch 2
 ├── Batch 3
 ├── ...
 └── Batch N

Instead of treating the entire dataset as one indivisible operation, the system processes manageable groups of records.

Each batch can be validated, written, and committed independently, depending on the consistency requirements of the import.

Batch size matters. Smaller batches generally reduce transaction duration and the amount of work that must be retried, but they can increase the number of database round trips. Larger batches may improve throughput, but they can consume more memory and take longer to process.

The right batch size depends on the database, record complexity, constraints, indexes, available resources, and acceptable recovery cost.

Batching creates a recovery point

Suppose:

Batch 1  → committed
Batch 2  → committed
...
Batch 47 → committed
Batch 48 → failed

We should not necessarily restart from the beginning.

If the earlier batches have been committed successfully, the system should be able to identify that progress and resume from the appropriate point.

However, batching alone does not guarantee resumability.

The application needs a reliable way to determine which records or batches have completed. A counter held only in application memory is insufficient because the process may restart and lose that information.

Depending on the import, progress can be tracked through:

A durable checkpoint recording the last successfully processed batch.
An import-job record containing the current status and progress.
Stable source identifiers that allow the application to reconcile existing records.
A combination of checkpoints and database state.

The checkpoint must accurately reflect committed work. If the system records a batch as complete before its database transaction commits, a failure could cause records to be skipped during recovery.

Conversely, if the database commits but the process crashes before updating the checkpoint, the batch may run again. That is why safe retries and idempotency remain important even when checkpoints are available.

The real problem is resumability

A long-running operation should answer:

What has completed?
What failed?
Where did the failure occur?
Can the operation safely resume?
Are records idempotent?
Can a partially completed batch be retried?
Can the system distinguish a failed operation from one that is still running?

These questions are more important than simply making the initial import work.

For a production import, I would also want to know the total number of records, how many have succeeded or failed, when the operation last made progress, and whether the failure is transient or requires intervention.

That information makes the operation observable and gives an operator a way to understand its state without reconstructing everything from application logs.

A useful design is to represent the import as a durable job with explicit states, such as:

PENDING
   ↓
RUNNING
   ├──→ COMPLETED
   ├──→ FAILED
   └──→ PAUSED / RETRYABLE

The exact states depend on the requirements. What matters is that the system can explain its current state and recover from interruption without relying on assumptions about what happened before the failure.

Network failure is normal

During one of the data-import workflows I worked through, a network connectivity failure occurred after many records had already been committed.

The useful lesson was not the specific error.

It was the architecture of the recovery problem.

A network failure does not necessarily tell us whether the database operation failed. In some situations, the client loses its connection after the database has committed a transaction but before the client receives confirmation.

The application may therefore be uncertain about the outcome.

Blindly repeating the operation can create duplicates or violate constraints. Assuming success can leave records missing.

This is why recovery logic should rely on durable database state, stable record identities, and safe retry behavior rather than treating every connection error as proof that nothing was committed.

A long-running operation must be designed with interruption in mind.

Idempotency

If we retry a batch, we should avoid creating duplicates or corrupting existing data.

That means the import needs a stable identity for each record or another mechanism for safely determining whether the record has already been processed.

For example, an imported personnel record might have a unique service number, while an organizational record might have a stable directory code. These identifiers can help the application reconcile incoming data with existing records.

In PostgreSQL, unique constraints provide an important database-level safeguard. Depending on the intended behavior, the import can use an insert, an upsert, or an explicit reconciliation step.

However, idempotency is not simply a matter of ignoring duplicate-key errors. The system must define what should happen when an existing record is encountered:

Should the existing record remain unchanged?
Should selected fields be updated?
Should conflicting values cause the import to fail?
Should the conflict be recorded for manual review?

The answer depends on whether the import is intended to create new records, synchronize an external source, or update an existing dataset.

The application and database should enforce the chosen behavior consistently.

Retrying the right unit of work

Another important consideration is deciding what to retry.

If a batch fails, retrying the entire import may waste time and repeat completed work. Retrying only the failed batch is often more efficient, provided the batch can be executed safely more than once.

A practical retry strategy should distinguish transient failures, such as temporary network interruptions or database unavailability, from permanent failures, such as invalid data or violated business rules.

Transient failures may justify a limited number of retries with exponential backoff and jitter. Permanent failures usually require correction rather than repeated execution.

For partially processed batches, the design must also establish the unit of atomicity. If each batch is committed as one transaction, a failed transaction should roll back its database changes. If records are committed individually, the retry mechanism must account for the records that succeeded before the failure.

This distinction affects how checkpoints are recorded and how much work recovery may need to repeat.

Final thought

A production data import is not just an insertion script.

It is a recoverable workflow.

Batching helps control the amount of work performed at once. Durable progress tracking makes recovery possible. Idempotency makes retries safer. Observability makes failures understandable.

Once operations become long-running, failure and resumption need to be considered part of the design rather than exceptional afterthoughts.

Designing Audit Trails for Systems That Matter

Tope Taiwo — Thu, 15 Oct 2026 05:00:00 +0000

Design audit trails that record who did what, when, and to which resource, supporting accountability in business-critical systems.

An audit log is not simply a table containing timestamps.

In systems involving inventory, approvals, assets and organizational responsibilities, knowing that something happened is not enough.

We also need to know what happened and who initiated it.

Auditability changes how a system is designed

Consider a requisition.

A useful audit trail may need to answer:

Who?
What?
When?
Which resource?
Which action?
What changed?

That is fundamentally different from simply logging:

Request completed successfully.

Audit and application logs are different

Application logs help us as engineers understand system behavior.

Audit records serve a different purpose.

For example:

Application log:
Database connection failed.

Audit record:
User approved requisition X.

The first helps operate the system.

The second records a business action.

Audit sensitive workflows

The NIS Store MIS contains workflows involving:

requisitions;
approvals;
inventory;
assets;
dispatch.

These are precisely the kinds of operations where traceability becomes important.

Do not record meaningless noise

Auditing everything can produce an enormous amount of information without creating useful accountability.

The better question is:

Which business actions must remain traceable?

Examples include:

creation;
approval;
rejection;
dispatch;
assignment;
status changes;
sensitive administrative changes.

Audit records should be difficult to misinterpret

An audit record should provide enough context to understand the action without requiring the reader to reconstruct the entire request from unrelated application logs.

This is especially important when an organization needs to investigate what happened later.

Validate audits records before persistence

It is germane to mention this because paying attention to details ensure systems are safer from attackers.

I mean Audit Input Validation (AIV).

Validating audits input before persisting in the database can help reduce attack vectors on core infrastructure especially the DATABASE.

Final thought

Auditability is not something that should be bolted onto an enterprise system at the end.

It should be considered when designing important business workflows.

If an operation matters to the organization, the system should make its history understandable.

DEV Community: Tope Taiwo

Payments Are a Workflow, Not Just an API Call

A payment is a workflow

Why the API response is not the entire story

Verify before granting access

Idempotency matters

Keep financial records separate from business effects

Payment architecture

Reconciliation is part of payment engineering

Final thought

Multi-Tenancy: What Changes When One Application Serves Many Organizations

What is multi-tenancy?

Tenant context

Data isolation

Authorization must include tenant context

Roles can also be tenant-specific

Tenant isolation extends beyond database queries

Testing tenant boundaries

Why this connects to my other work

Final thought

Building Frontends That Respect Backend Architecture

Separate server state from client state

React Query

Zustand

API clients

Respect the backend contract

Next.js

Organize around features and responsibilities

Final thought

Performance Engineering: Measure Before You Optimize

An assumption is not a benchmark

Measure the right thing

Establish a baseline first

Measure the actual workload

Database performance

Understanding Node.js performance

Optimize the bottleneck, not the architecture diagram

Benchmarking belongs in engineering

Final thought

Going Deeper Into Node.js: Understanding the Runtime Behind the Framework

The framework is not the runtime

Why the event loop matters

Understanding the event loop beyond async and await

Asynchronous does not mean parallel

CPU-heavy work needs a different strategy

Streams matter too

Memory management and garbage collection

Networking and connection management

Process behavior and graceful shutdown

Framework knowledge is not enough

Measure before optimizing

Final thought

Testing Backend Architecture, Not Just Functions

Different tests answer different questions

Test business logic independently

Integration tests still matter

Test the boundaries between layers

Test architecture through dependency direction

Failure testing

Test transactional consistency

Contract testing for replaceable infrastructure

End-to-end tests should protect critical workflows

Avoid testing implementation details

Coverage is a signal, not the objective

Final thought

Transactions, Events and the Problem of Reliable Side Effects

The dual-write problem

Transactions do not automatically solve messaging

The transactional outbox pattern

A practical implementation

The publisher introduces another failure boundary

Consumer idempotency still matters

Polling versus change data capture

Polling

Change data capture

Event ordering and concurrent publishers

Retries, dead-lettering, and operational recovery

This connects to previous decisions

Final thought

Designing Background Jobs Without Infrastructure Coupling

Understanding the event loop beyond `async` and `await`