DEV Community: Temitope Bamidele

Observability Engineering in Production Systems: Structured Logging, Metrics, and Distributed Tracing at Scale (Part 3)

Temitope Bamidele — Sun, 12 Apr 2026 18:46:15 +0000

How Observability Engineering Cut Incident Response Time by 85% in Production (Part 3)

Part 3 of 3: Security, Metrics, and Full-Stack Observability

This article covers two things most logging setups skip until they cause a problem: the compliance and security constraints that govern what you are allowed to log, and how structured logs connect to metrics and distributed traces to produce an observability system where incidents are detected in minutes rather than hours. It assumes your logs are structured JSON, centralised into a queryable store like ELK or Loki. If you're reading this as part of the series, that foundation is covered in Parts 1 and 2.

8. Security and Compliance in Logging

Logs are a security surface. If you're building in any regulated industry — fintech, healthcare, any platform handling personal data — your logs are subject to data protection obligations. In Nigeria, the NDPR (Nigeria Data Protection Regulation) restricts how personal data is stored and processed. In the UK, GDPR applies. In both cases, personal data appearing in log files creates compliance exposure: logs are often retained for months, frequently shipped to third-party aggregation services, and sometimes not encrypted at rest.

The risks here are not abstract. Bearer tokens and API keys: if your middleware serialises request headers and you haven't configured explicit redaction, they are in your log files right now. This is a slow-burn exposure — logs get retained for months, shipped to third-party aggregation services, and nobody typically audits them for credential leakage until there's a reason to. We auto-redact a fixed list of paths at the Pino layer precisely because "developers will remember not to log sensitive fields" is not a security policy.

Card data and BVN numbers need specific attention even when you're not storing them. Platforms handling Nigerian identity verification commonly pass BVNs through consent flows, and a debug log that serialises the full request body can silently capture and retain them. Add explicit redaction for any field that could contain a card number, account number, or national identity number. While you're at it, verify that your payment provider isn't including raw card data in webhook payloads — older integration patterns occasionally do depending on account configuration.

For user records, the working principle is: log enough to diagnose, not enough to reconstruct the person's identity:

// log-sanitiser.ts
export function sanitizeForLog(user: User): Record<string, unknown> {
    return {
        userId: user.id,
        // Mask email: show domain and first two chars
        email: maskEmail(user.email),
        // Mask phone: show last 4 digits only
        phone: user.phone?.replace(/\d(?=\d{4})/g, "*"),
        tenantId: user.tenantId,
        roles: user.roles
    };
}

function maskEmail(email: string): string {
    const [local, domain] = email.split("@");
    const visibleChars = Math.min(2, local.length - 1);
    const masked =
        local.slice(0, visibleChars) + "*".repeat(local.length - visibleChars);
    return `${masked}@${domain}`;
}

// Usage
this.logger.log({
    event: "user.payment.initiated",
    user: sanitizeForLog(user),
    amount: dto.amount,
    reference: dto.reference
});

Log retention policy. Define how long logs are retained and enforce it with index lifecycle management. The retention window isn't arbitrary — for a platform processing payments in Nigeria, NDPR requires that you can demonstrate the lawful basis for data processing and respond to data subject requests, which practically means payment-related logs need to be queryable for at least 12 months. The ILM policy below uses 365 days for that reason: long enough to cover a full annual audit cycle. Application debug logs have no compliance value after the event they document has passed; 7–30 days is sufficient and aggressive deletion controls storage costs.

// Elasticsearch ILM policy for payment logs
{
    "policy": {
        "phases": {
            "hot": {
                "min_age": "0ms",
                "actions": {
                    "rollover": {
                        "max_size": "10GB",
                        "max_age": "7d"
                    }
                }
            },
            "warm": {
                "min_age": "30d",
                "actions": {
                    "shrink": { "number_of_shards": 1 },
                    "forcemerge": { "max_num_segments": 1 }
                }
            },
            "delete": {
                "min_age": "365d",
                "actions": { "delete": {} }
            }
        }
    }
}

Log access control. Your log aggregation system should have the same access control standards as your production database. Everyone shouldn't have unrestricted query access to payment logs. A developer debugging a UI issue doesn't need to query logs that contain donor transaction details. Implement role-based access in Kibana or Grafana and review it as part of your access control audits.

9. From Logs to Observability: The Full Picture

Logs are one piece of a larger system, and the most important thing to understand about them is that they're forensic by nature — you reach for them when you already know something is wrong. That makes them poor for incident detection. If your only visibility is logs, you find out about problems when someone notices them.

Metrics close the detection gap. Request rate, error rate, latency percentiles, queue depth — aggregated time-series that Prometheus scrapes every few seconds and evaluates against alert thresholds. A webhook error rate crossing 5% fires an alert within minutes of the condition developing, not after a support ticket lands. Metrics don't tell you why; they tell you that.

Distributed tracing closes the attribution gap. When latency spikes and neither logs nor metrics can tell you which operation is responsible, traces show the breakdown across the entire call graph: mongoose.find took 12ms, redis.get took 3ms, paystack.initializeTransaction took 773ms. Without tracing you're sampling log timestamps and doing arithmetic. With it the slow operation is visible in seconds.

The three are genuinely complementary, not redundant. Metrics for detection, logs for diagnosis, traces for attribution. The practical implication: don't build your rate and volume alerting on log queries — it's the wrong tool for the job. Querying logs to derive error rates is expensive and slow. Build alerting on metrics, use logs to understand what the alerts are telling you, use traces when you need to know where time went.

When Log Queries Are the Right Tool

There is one class of alerting where a log query is the right approach: structural pairing checks. A structural pairing check — for example, querying every 15 minutes for webhook.processing.started events without a corresponding completed or exhausted event — is not a rate query. It is a pairing check, and there is no clean Prometheus equivalent without engineering additional counters to track in-flight job state. For structural invariants of this kind, a scheduled log query run against the aggregation store is the appropriate tool. The distinction is: metrics for "how much", log queries for "did every A produce a B".

Metrics with Prometheus

Prometheus is the standard choice for metrics collection in NestJS services, with Grafana for dashboards. The nestjs-prometheus package makes it straightforward to expose metrics from a NestJS service:

// metrics.service.ts
import { Injectable } from "@nestjs/common";
import { InjectMetric } from "@willsoto/nestjs-prometheus";
import { Counter, Histogram } from "prom-client";

@Injectable()
export class MetricsService {
    constructor(
        @InjectMetric("http_requests_total")
        private readonly requestCounter: Counter<string>,

        @InjectMetric("http_request_duration_seconds")
        private readonly requestDuration: Histogram<string>, // buckets registered at module level: [0.1, 0.25, 0.5, 0.75, 1.0, 1.5, 2.0, 2.5, 5.0]

        @InjectMetric("payment_webhooks_total")
        private readonly webhookCounter: Counter<string>,

        @InjectMetric("payment_settlements_total")
        private readonly settlementCounter: Counter<string>
    ) {}

    recordRequest(method: string, route: string, status: number): void {
        this.requestCounter.inc({ method, route, status: String(status) });
    }

    recordRequestDuration(
        method: string,
        route: string,
        durationMs: number
    ): void {
        this.requestDuration.observe({ method, route }, durationMs / 1000);
    }

    recordWebhook(
        provider: string,
        status: "received" | "failed" // 'received' at HTTP ingestion; 'failed' on exhausted retries
    ): void {
        this.webhookCounter.inc({ provider, status });
    }

    recordSettlement(provider: string, status: "success" | "failed"): void {
        this.settlementCounter.inc({ provider, status });
    }
}

The most important Prometheus alert we run:

# alert-rules.yml
groups:
    - name: payment-reliability
      rules:
          - alert: WebhookSettlementDelta
            expr: |
                increase(payment_webhooks_total{status="received"}[30m])
                -
                increase(payment_settlements_total{status="success"}[30m]) > 2
            for: 5m
            labels:
                severity: critical
            annotations:
                summary: "Webhook-to-settlement delta exceeds threshold"
                description: >
                    More webhooks are being received than settlements recorded.
                    Possible silent settlement failures. Current delta: {{ $value }}

          - alert: HighWebhookErrorRate
            expr: |
                (
                  rate(payment_webhooks_total{status="failed"}[5m])
                  /
                  rate(payment_webhooks_total{status="received"}[5m])
                ) > 0.05
                and
                rate(payment_webhooks_total{status="received"}[5m]) > 0
            for: 2m
            labels:
                severity: warning
            annotations:
                summary: "Webhook failure rate exceeds 5%"

The threshold choices here are deliberate, not arbitrary. The delta of 2 (rather than 1) accounts for webhooks legitimately in-flight through the queue — received and enqueued but whose settlement write hasn't completed yet, a normal transient state that resolves within seconds. A sustained delta of 2+ over 5 minutes is not queue lag; it's a failure. The 5% error rate threshold was calibrated against two months of baseline data: a healthy payment service typically sits below 1% webhook errors, so 5% represents a 5x deviation from baseline — unambiguously abnormal, not a provider blip. Anything tighter produced false positives during transient provider degradation windows.

HighWebhookErrorRate is warning rather than critical because a 5% failure rate against a Bull queue with retries configured means most of those failures will self-recover within the retry window. WebhookSettlementDelta is critical because it fires only when failures have persisted through retries and settlements are already unaccounted for — a state that requires immediate human action, not automated recovery.

The webhook-to-settlement delta alert is the metric equivalent of the reconciliation log check. Metrics fire faster — within minutes via Prometheus alertmanager — than a cron-based log analysis. Logs then give you the detail to understand why.

Distributed Tracing with OpenTelemetry

OpenTelemetry is the vendor-neutral standard for distributed tracing. Setting it up in NestJS instruments your HTTP requests, database calls, and external HTTP calls automatically:

// tracing.ts — initialise before anything else
import { NodeSDK } from "@opentelemetry/sdk-node";
import { OTLPTraceExporter } from "@opentelemetry/exporter-trace-otlp-http";
import { getNodeAutoInstrumentations } from "@opentelemetry/auto-instrumentations-node";
import { PinoInstrumentation } from "@opentelemetry/instrumentation-pino";

const sdk = new NodeSDK({
    traceExporter: new OTLPTraceExporter({
        url: process.env.OTEL_EXPORTER_OTLP_ENDPOINT
    }),
    instrumentations: [
        getNodeAutoInstrumentations({
            "@opentelemetry/instrumentation-http": { enabled: true },
            "@opentelemetry/instrumentation-express": { enabled: true },
            "@opentelemetry/instrumentation-mongoose": { enabled: true },
            "@opentelemetry/instrumentation-ioredis": { enabled: true }
        }),
        new PinoInstrumentation() // Injects traceId + spanId into every Pino log entry automatically
    ],
    serviceName: process.env.SERVICE_NAME
});

// start() is non-blocking. If the OTLP collector is unreachable, spans are dropped
// rather than buffered indefinitely — tracing degrades gracefully without affecting
// the application. OTEL_SDK_DISABLED=true disables it entirely in environments
// where you don't want the overhead (e.g. local unit test runs).
sdk.start();

With this in place, every HTTP request generates a trace with spans for each downstream operation. When a donation endpoint takes 800ms and you can't tell if it's the database, the cache, or the payment provider's initialization call, the trace shows the breakdown: mongoose.find took 12ms, redis.get took 3ms, paystack.initializeTransaction took 773ms. The slow operation is immediately visible without any additional instrumentation.

Connecting the Three Pillars

The correlation ID is what allows logs and traces to link back to the same request. Because PinoInstrumentation is registered in tracing.ts, OpenTelemetry injects traceId and spanId into every Pino log entry automatically — no per-call code required. Every log entry emitted during an active trace will include them:

{
    "level": "info",
    "timestamp": "2025-11-14T02:41:33.421Z",
    "service": "payment-service",
    "event": "donation.settled",
    "requestId": "f3a2b1c0-9e8d-4b7a-a6c2-d3e4f5a6b7c8",
    "reference": "PAY-abc123",
    "amount": 50000,
    "traceId": "4bf92f3577b34da6a3ce929d0e0e4736",
    "spanId": "00f067aa0ba902b7"
}

In Grafana, this allows you to navigate from a slow trace directly to the correlated log entries, or from an error log directly to the distributed trace that shows the full timing breakdown. This linkage is what makes incident investigation dramatically faster — you can pivot between views without manually correlating IDs across two separate tools.

A production payment infrastructure dashboard typically combines all three:

Top row: Webhook ingestion rate vs settlement rate, per provider — the gap between these two lines is the most important signal on the board. A widening gap means money is moving through the payment network but not landing in donor accounts.
Second row: Bull queue depth and active worker count — queue depth climbing while workers are idle means a worker process is hung or has lost its Redis connection.
Third row: p95 payment initialization latency per provider, with expected-range reference lines. Flutterwave routinely spikes to 1.2s at peak; anything consistently past 2.5s triggers investigation.
Bottom row: Failed job count vs retry budget remaining. When the retry budget on a webhook batch approaches zero and the dead-letter queue starts filling, that's a critical alert before a single settlement is permanently lost.
Drill-down: Click any error log → navigate to the correlated distributed trace → see the exact timing breakdown across the call graph.

Mean time to detection followed a four-stage progression. Before ELK — the period of the opening incident — investigation meant manually correlating events by timestamp across services; that incident ran silently for over two hours before the failure was located. After ELK was deployed but before any proactive alerting existed, detection dropped to 18 minutes (median, measured from incident start to first alert): an engineer still had to notice the anomaly in Kibana, but the information was there and queryable. After post-incident log-based structural pairing alerts were introduced — scheduled queries checking every 15 minutes for webhook.processing.started events without a matching completed or exhausted event — the three subsequent anomalies were caught in an average of eight minutes; the alerts fired automatically rather than waiting for someone to look. After the full metrics dashboard went live with Prometheus alertmanager: 3 minutes. Each layer reduced detection time not by doing the same thing faster, but by eliminating the manual step that preceded it.

10. What I Would Tell Someone Building This Today

The honest answer is: start earlier than you think you need to.

Structured logging and correlation IDs feel like overhead when your system is small. The logs are short, you remember what every code path does, and you can usually find issues by reading raw output. Then the system grows — a second service, async queue processing, a few replicas behind a load balancer — and suddenly the things that used to take five minutes to debug take two hours. Not because the bugs got harder. Because the information you need is spread across ten files with nothing tying it together.

Retrofitting structured logging onto a running system is genuinely painful. Every Kibana query you've written against free-text messages breaks. Dashboards need rebuilding. You do this work under pressure because the retrofit almost always happens in the immediate aftermath of the incident that exposed the gap. The upfront investment is a few hours. The retrofit is a week you don't have.

On correlation IDs: if you take nothing else from this article, propagate a requestId through every log entry and every async job payload. The ability to query "show me everything related to this request" and get a complete picture — from HTTP receipt through queue processing through settlement — is the single biggest reduction in investigation time I've seen. I've watched engineers spend thirty minutes getting context during an active incident because there was no way to isolate a single request's trail. With correlation IDs that's a ten-second Kibana query.

Log levels need more rigour than most teams give them. That alert fatigue period developed over months without anyone making a deliberate decision to accept noisy alerts. Individual developers logged recoverable exceptions at ERROR because that's the level that felt right for something going wrong. The fix required a deliberate audit and a team conversation about what ERROR actually means for our system. Worth having that conversation before you need it.

Build at least minimal metrics alerting early. Logs are forensic; they don't tell you something is wrong until you look at them. A Prometheus alert with a webhook-settlement delta rule would have caught the incident within minutes of the gap opening. Without it, the failure ran silently for over two hours and was only found through manual investigation — cross-referencing an external dashboard against database records with no shared identifier to work from.

One thing that gets skipped even on experienced teams: run a log scan before shipping to production. Pull any ten log lines from a recent staging run and read them. You'll find a debug log that serialises a full request body, including a field someone named password or bvn in a feature branch that never got cleaned up. The Pino redact config catches explicit paths; it doesn't catch fields you didn't know existed.

One thing worth stating explicitly: no observability system sees everything. The four-stage mean time to detection improvement assumes the aggregation pipeline is healthy — when our Elasticsearch cluster was degraded during one incident, we were back to Paystack's dashboard and raw database queries. Distributed tracing covers what you've instrumented; a slow call from a third-party library won't produce spans unless that library exposes OTel hooks. Prometheus alerting has a minimum resolution equal to its scrape interval. And structural observability doesn't detect semantic errors: a webhook that processes successfully by every measurable signal but records the wrong amount due to a currency conversion bug will not trigger any of the alerts described here. Understanding the edges of your visibility — what your system genuinely cannot see — is as important as understanding what it can.

And finally: treat your aggregation infrastructure as a production system. It needs its own health monitoring, capacity planning, and a recovery runbook. I once debugged a payment failure using only Paystack's dashboard and raw database queries because our Elasticsearch cluster was degraded during the incident. It was workable, but slow and uncomfortable in a way that felt completely avoidable. The cluster now has its own monitoring and a documented recovery procedure. That took half a day to write and has been used twice since.

Wrapping Up

There's a reason that observability is described as the difference between operating a system and flying blind. In production, you can't know in advance what questions you'll need to answer. You can't predict which combination of services will produce a failure you haven't seen before. What you can do is build a system where every event is recorded with enough structure to answer those questions retroactively — quickly, precisely, without reconstructing context from memory at 11pm.

Production systems differ in scale, traffic profile, and failure modes. What they share — or should — is a logging discipline that was designed rather than accrued. Across three production incidents in a six-month window, every one was traced to root cause and resolved without escalation. Mean time to resolution dropped from well over an hour to under fifteen minutes for the most common incident classes. The time it takes a new engineer to get oriented during an active incident — what used to be thirty minutes of reading through disconnected logs across multiple servers — is now a ten-second Kibana query.

Logging isn't a feature. It's the operational foundation everything else runs on. If you can't see what your system is doing, you don't really know what it's doing.

Build the visibility first.

Observability Engineering in Production Systems: Structured Logging, Metrics, and Distributed Tracing at Scale (Part 2)

Temitope Bamidele — Sun, 12 Apr 2026 18:42:46 +0000

How Observability Engineering Cut Incident Response Time by 85% in Production (Part 2)

Part 2 of 3: Centralized Logging and Async Pipelines

This article covers where structured logs go once they leave your application: centralised log aggregation that makes them queryable across services and time, with the async pipeline patterns that keep distributed systems observable. It assumes two foundational practices are in place: logs are structured JSON records with consistent event naming, and each request carries a correlation ID that propagates through every downstream operation. If you're reading this as part of the series, those are covered in Part 1. The full story of the Helpa production incident — a webhook regression that ran silently for over two hours — is reconstructed here through the logs that eventually exposed it.

5. Centralized Logging: Aggregating the Signal

Individual log files per service, written to disk, are not a logging system. They're a precondition for one. A log that lives only on the server that generated it is useless the moment you have more than one server, useless when that server restarts, and useless in practice even on a single server because reading raw files under incident pressure is slow and error-prone.

Centralized logging means all logs from all services flow to a single queryable store. Two architectures cover most production use cases:

ELK Stack (Elasticsearch, Logstash, Kibana)

Elasticsearch is a distributed search and analytics engine optimized for full-text and structured queries. Logstash (or the lighter Filebeat) collects and ships logs from your services. Kibana provides the query interface and dashboard layer.

The pipeline: your service writes JSON logs to stdout → a Filebeat sidecar ships to Logstash → Logstash parses, enriches, and routes to Elasticsearch → Kibana queries Elasticsearch. Two non-obvious decisions live in the Logstash filter:

# logstash.conf — filter block
filter {
  if [fields][service] {
    mutate { add_field => { "[@metadata][index]" => "%{[fields][service]}" } }
  } else {
    mutate { add_field => { "[@metadata][index]" => "unknown" } }
  }
  # Re-parse the app timestamp into @timestamp so Kibana time-range queries
  # reflect event time, not ingest time.
  date {
    match => ["[app][timestamp]", "ISO8601"]
    target => "@timestamp"
  }
}

With this configuration, logs-payment-service-2025.11.14 is a distinct Elasticsearch index. You can query across all indexes with logs-*, or scope to a specific service for faster queries.

Loki + Grafana

Loki is the lighter-weight alternative, designed for label-based log querying rather than full-text indexing. It's significantly cheaper to operate at scale because it doesn't index the content of log messages — only the labels (metadata like service, environment, host). Content is queried by filtering indexed log streams rather than full-text search.

The trade-off: Loki's LogQL query language is less expressive than Elasticsearch's query DSL for complex analytical queries. For operational use cases — "show me all ERROR events from the payment service in the last hour" — Loki is fast and cheap. For complex analytics — "what was the median time between webhook receipt and settlement for donations over ₦100,000 last month" — Elasticsearch is more capable.

On lower-volume, cost-sensitive workloads Loki is the pragmatic choice — you're not indexing log content, only labels, so storage costs stay manageable as the system grows. On systems with higher query complexity requirements — payment audit trails, multi-dimensional analytics — Elasticsearch is the more capable tool.

Promtail is the agent that collects and ships logs to Loki. The non-obvious decision in its pipeline config is what to promote to a Loki label and what not to:

# promtail-config.yml — pipeline_stages
pipeline_stages:
    - json:
          expressions:
              level: level
              service: service
    - labels:
          level: # low cardinality — safe as a Loki stream label
          service: # low cardinality — safe as a Loki stream label
          # requestId is intentionally NOT a label: it is a UUID unique per request
          # and would create millions of distinct Loki streams, exhausting memory.
          # Query it as a parsed field instead: | json | requestId="<value>"

With labels extracted and indexed, this Grafana Loki query retrieves all error-level log entries for the payment service from the last two hours and returns them in chronological order:

{service="payment-service", level="error"} | json | line_format "{{.timestamp}} {{.event}} {{.requestId}} {{.message}}"

6. Logging in Distributed Systems and Async Pipelines

The hardest logging problems are not in your HTTP request handlers. They're in the places that operate outside the standard request/response cycle: webhook processors, job queues, cron jobs, event bus consumers.

These components have no caller waiting for a response. If they fail silently, nobody knows. If they fail noisily but without context, you know something is wrong but not what or why or for whom.

The Webhook Black Hole

Webhooks are one of the highest-risk surfaces in any payment system precisely because they're the most consequential and the least observed. A webhook represents money in motion. A webhook that fails silently means a settlement that never happened, a campaign that didn't receive funds it's owed, a donor record that was never created.

The pattern I use consistently across both systems:

// webhook.processor.ts
@Process('process-webhook')
async handleWebhook(job: Job<WebhookJobData>): Promise<void> {
  const { payload, meta } = job.data;
  const startTime = Date.now();

  this.logger.log({
    event: 'webhook.processing.started',
    requestId: meta.requestId,
    provider: payload.provider,
    reference: payload.reference,
    jobId: job.id,
    attempt: job.attemptsMade,
  });

  try {
    await this.webhookService.process(payload, meta.requestId);

    this.logger.log({
      event: 'webhook.processing.completed',
      requestId: meta.requestId,
      reference: payload.reference,
      duration: Date.now() - startTime,
      jobId: job.id,
    });

  } catch (error) {
    const isLastAttempt = job.attemptsMade >= (job.opts.attempts ?? 1) - 1;
    const logContext = {
      requestId: meta.requestId,
      reference: payload.reference,
      error: error.message,
      stack: error.stack,
      jobId: job.id,
      attempt: job.attemptsMade,
    };

    if (isLastAttempt) {
      this.logger.error({ event: 'webhook.processing.exhausted', ...logContext });
    } else {
      this.logger.warn({ event: 'webhook.processing.failed', ...logContext });
    }

    throw error; // Re-throw so Bull marks the job for retry
  }
}

Notice webhook.processing.exhausted as a distinct event from webhook.processing.failed. A failed job that will be retried is a recoverable situation — WARN level, note and move on. A job that has exhausted all retries is a permanently lost webhook — ERROR level, alert immediately, requires manual intervention.

Logging Cron Job Behavior at the Right Granularity

Cron jobs that check every pending transaction or trigger reconciliation runs are critical to payment system integrity. They also generate enormous log volume if you log at the wrong granularity:

// reconciliation.cron.ts
@Cron('0 */6 * * *') // Every 6 hours
async runReconciliation(): Promise<void> {
  const runId = `run_${uuidv4().split('-')[0]}`;
  const startTime = Date.now();

  this.logger.log({
    event: 'reconciliation.run.started',
    runId,
    triggeredBy: 'cron',
  });

  const staleTransactions = await this.transactionService
    .findStalePending();

  if (staleTransactions.length === 0) {
    this.logger.log({
      event: 'reconciliation.run.completed',
      runId,
      resolved: 0,
      duration: Date.now() - startTime,
    });
    return;
  }

  let resolved = 0;
  let failed = 0;

  for (const tx of staleTransactions) {
    try {
      await this.reconciliationService.process(tx, runId);
      resolved++;
    } catch (error) {
      failed++;
      this.logger.error({
        event: 'reconciliation.transaction.failed',
        runId,
        transactionId: tx.id,
        reference: tx.reference,
        staleSince: tx.updatedAt,
        error: error.message,
      });
    }
  }

  this.logger.log({
    event: 'reconciliation.run.completed',
    runId,
    total: staleTransactions.length,
    resolved,
    failed,
    duration: Date.now() - startTime,
  });
}

The runId stitches all log entries from a single reconciliation run together, exactly as requestId does for HTTP requests. This means when reconciliation partially fails — some transactions recover, some don't — you can filter by runId in Kibana and see the complete picture of that specific run.

7. The Incident That Changed Everything

The Helpa incident is worth unpacking in full, because the root cause is the kind of thing that catches experienced teams — not junior ones.

Earlier in the project, the webhook receiver passed the raw Paystack payload directly onto the queue. During a broader normalisation effort, this was refactored: the receiver now mapped the inbound payload to a typed internal event object before enqueuing, so the queue processor would always handle a consistent internal schema rather than Paystack's raw shape. As part of that normalisation, the internal field previously named donorId was renamed to customerId to align with the rest of the platform's identity model. The receiver was updated. The processor — in a separate module, decoupled across the queue boundary — was not.

The processor extracted the identity field using optional chaining: const donorId = job.data?.donorId. After the deployment, donorId was no longer present in the enqueued payload; the field was now customerId. Optional chaining resolved silently to undefined — valid TypeScript, no error, nothing flagged.

Months earlier, a guard had been added near the top of the processing block. It had been added as a precaution against a specific edge case during early development, had never fired in any meaningful volume, and carried no log alongside it because the path was considered unreachable in normal operation:

const donorId = job.data?.donorId; // undefined after rename — no compile error
if (!donorId) {
    return;
} // no log — fires for every job post-deployment

When the field rename made donorId absent in every enqueued job, that guard became the first thing the processor hit, and it fired silently for every one of them.

Paystack fires a webhook. The receiver maps it to the internal schema, enqueues the job with customerId set correctly. The queue processor runs, reads job.data?.donorId, gets undefined, hits the unlogged guard, and returns. No exception, no error log, no dead-letter entry. No donation ledger entry. No wallet credit. Bull marks the job completed.

The donor sees a completed payment on their bank statement. The campaign owner sees nothing.

When I looked at what we had in the logs:

{ "level": "info", "timestamp": "2023-11-09T14:22:17.341Z", "service": "payment-service", "event": "webhook.received", "reference": "PAY-abc123", "requestId": "f3a2b1c0-9e8d-4b7a-a6c2-d3e4f5a6b7c8" }
{ "level": "info", "timestamp": "2023-11-09T14:22:17.389Z", "service": "payment-service", "event": "webhook.job.enqueued", "reference": "PAY-abc123", "customerId": "usr_7f3a1b2c", "jobId": "4821", "requestId": "f3a2b1c0-9e8d-4b7a-a6c2-d3e4f5a6b7c8" }

And then nothing. No processing entry of any kind. The processor had run, exited cleanly from Bull's viewpoint, and left no trace — because the guard that fired had no accompanying log call. It was a return statement with no record that it had ever executed. A job that Bull considered completed was indistinguishable in the logs from a job that had successfully processed a donation.

The normalisation refactor wasn't wrong — unifying the internal schema was the right call. The guard wasn't wrong either; guarding against missing identity on a payment processor is sensible. What was wrong was the combination: a cross-module rename across a queue boundary, with no integration test covering the end-to-end path from enqueue to processDonation() being called, and an unlogged early return that had been written off as unreachable. TypeScript can enforce payload contracts across a queue boundary — but only if you explicitly share a typed Job<WebhookJobData> generic between the producer and consumer. The processor was a legacy module that predated the team's move toward shared payload generics; it had been working correctly with job.data.donorId typed loosely for months. The normalisation refactor introduced the shared WebhookJobData type for new code and updated the receiver, but didn't backfill the processor. Optional chaining on a loosely typed field is structurally valid code regardless of whether the field is present at runtime, so the rename produced no compile error and the gap in the refactor's scope went unnoticed.

This is worth naming as a general principle: silent early returns in critical processing code — even well-intentioned defensive ones — are an observability hazard. They are not exception paths, so they bypass error handlers. They are not intentional exits, so they're easy to forget to log. They tend to be written during early development when their execution seems implausible. And when a schema change, a field rename, or an unexpected edge case makes them suddenly reachable, they become invisible. Every exit from a payment processor — clean or otherwise — should leave a record in the logs.

In hindsight the observability gaps were clear: no entry log at the top of the processor before any branching, no log on the early return that had become unexpectedly reachable, no reconciliation check linking webhook receipt to an expected settlement, no paired-event alerting to catch a started without a corresponding completed. None of these were hard to build. The system had simply grown to a complexity where their absence became expensive.

After the incident, every async processor was updated to emit a started log unconditionally at the point of entry — before any branching, before any validation — so that code paths exiting early still leave a visible trace. Alerting runs a query every 15 minutes: if there are more webhook.processing.started events than webhook.processing.completed plus webhook.processing.exhausted events in the last 30 minutes — accounting for the processing time window — an alert fires. This means a webhook that starts processing but never reaches a terminal state is caught within 15 minutes.

Additionally, the reconciliation cron was updated to log not just stale pending transactions but the delta against expected settlements: how many webhooks fired today, how many settlements recorded, how many are unaccounted for. The log entry looks like:

{
    "event": "reconciliation.integrity.summary",
    "runId": "run_9c3d2f1a",
    "period": "last-24h",
    "webhooksFired": 847,
    "settlementsRecorded": 847,
    "delta": 0,
    "status": "clean"
}

When delta is non-zero, the alert fires and the individual unreconciled references are listed in the same log entry.

The outcome: In the six months following the incident, three separate payment processing anomalies were caught by this alert — two caused by one of our payment gateways returning HTTP 200 responses with a status: false body during a transient degradation period (our processor was checking only for non-2xx HTTP errors and treated those responses as successful, leaving settlements silently unqueued), and one caused by a deployment that briefly broke queue worker connectivity. All three were caught and resolved without donor impact. Average detection time: eight minutes. Average resolution time: twelve minutes.

Compare that to the original incident, which ran silently for over two hours and was only found through manual investigation — cross-referencing Paystack's dashboard against database records with no shared identifier to work from. The automated reconciliation cron was no help — it looked for stale pending transactions, but no transaction records existed because the processor had returned before processDonation() was ever reached. Had that manual investigation not located the gap, the accounting team would have caught it eventually by reconciling Paystack's settlement reports against the donation ledger — but that is an audit cycle measured in days, not a detection mechanism. The structural pairing alert and delta reconciliation introduced after the incident close that gap: the same failure is now caught within minutes, automatically, without waiting for a campaign owner's vigilance or a finance team's next audit run.

Centralised log aggregation, async pipeline instrumentation, and structural alerting change the nature of incident response. The Helpa incident described in this article — a webhook regression that ran silently for over two hours — would have been detected in under eight minutes with the structural pairing alert introduced at the end. The logs existed; what was missing was the infrastructure to make them queryable and the alerting to make the signal automatic. Across the three subsequent incidents in the six months that followed, average detection time was eight minutes and average resolution time was twelve.

The two remaining pieces — what you are allowed to log in regulated environments, and how logs connect to metrics and traces for proactive detection — are covered in Part 3 if you're continuing the series.

Observability Engineering in Production Systems: Structured Logging, Metrics, and Distributed Tracing at Scale (Part 1)

Temitope Bamidele — Sat, 11 Apr 2026 23:14:37 +0000

How Observability Engineering Cut Incident Response Time by 85% in Production (Part 1)

Part 1 of 3: Structured Logs and Correlation IDs

In late 2023, Helpa's payment infrastructure went quiet in the worst possible way.

Helpa is a crowdfunding platform — people use it to raise money for surgeries and emergency expenses. A donor completed a payment on Paystack. The gateway returned a success response. The webhook fired. And then — nothing. No settlement. No ledger entry. No wallet credit. The campaign stayed at its previous total as if the donation had never happened. The donor had been charged. The campaign owner was waiting.

What followed was two hours of reconstruction: cross-referencing Paystack's dashboard against our database records, reading through raw log files on two different servers, trying to piece together a request timeline with no shared identifier across the output. We found it eventually — a regression in the webhook queue processor that caused jobs to exit silently without processing — but nothing in our logging surface had made the failure visible while it was happening. By the time we knew there was a problem, it had been running for over two hours.

I want to be precise about what the actual failure was, because the easy reading is "your logging was bad." It wasn't, exactly. Helpa started as a single-service deployment with modest transaction volume. The logging we had was adequate for that: structured enough to be readable, centralised enough that one person could scan the output and understand what was happening. That worked for a long time.

What changed was the system. By 2023 Helpa was processing hundreds of webhook events on a normal day, with traffic spiking 5–8x during successful campaign shares. We were running async queue processing across three payment providers, a reconciliation cron, and multiple replicas behind a load balancer. The observability model hadn't kept pace with that complexity. We still had the logging practices of a simpler system applied to infrastructure that had quietly become much harder to observe. The gap wasn't negligence — it was that we hadn't felt the operational friction of the old model badly enough until we were well into an active incident.

That incident was the forcing function for a complete rethink of how I approach observability — not as a debugging utility bolted on after the fact, but as an architectural discipline that evolves alongside the system. This article documents what I built, the reasoning behind each decision, and what it changed.

One conclusion I want to flag upfront, because it took time to articulate clearly: logs, metrics, and traces are not three ways of doing the same thing. They answer structurally different questions — and understanding when to reach for each one, rather than defaulting to whichever is familiar, is the operational discipline this article is really about. Getting that wrong doesn't just make your system harder to operate — it shapes which failures are visible and which ones aren't.

1. The Illusion of Logging

Most backend engineers have logging. Few have useful logging.

The first form of logging is the console.log. It's fast to write, and it's enough when you're running code on your own machine and can watch the output scroll past in real time. It communicates nothing to a production system. console.log('payment processed') tells you something happened. It doesn't tell you which user triggered it, what the payment reference was, what the amount was, whether it was the first time the webhook fired or the third, or which instance of your API server handled it when you're running three replicas behind a load balancer.

The second, more dangerous form of logging is the one that looks like real logging but isn't. This is where teams use a proper logging library — Winston, Pino, Bunyan — but configure it to write plain text messages to a file that nobody reads until something breaks. The messages have timestamps and log levels, which creates the impression of a proper system. But the content is still unstructured narrative: Payment webhook received for reference PAY-12345 written as a free-text string rather than as a queryable record. When an incident happens and you need to find all events related to a specific user's payment across multiple services, you're running grep and hoping the reference number appears consistently in every relevant log line.

Here's the thing: a log you cannot query isn't a log — it's an audit trail that only works for the events you already know to look for. You need to answer questions you haven't thought of yet, correlate events across services you didn't know were involved, and do all of it under pressure at 1am when a campaign owner's surgery fundraiser is stalled.

That requires a fundamentally different approach to what a log entry is.

2. Structured Logging: The Foundation

A structured log entry is a record, not a sentence. Every field is named, typed, and independently queryable. The difference is immediate once you see it:

Unstructured:

[2025-11-14 02:41:33] INFO Payment webhook received for reference PAY-abc123, amount 50000, result: success

Structured:

{
    "level": "info",
    "timestamp": "2025-11-14T02:41:33.421Z",
    "service": "payment-service",
    "event": "webhook.received",
    "provider": "paystack",
    "reference": "PAY-abc123",
    "amount": 50000,
    "currency": "NGN",
    "campaignId": "camp_8f2a1c",
    "userId": "usr_3b9d2e",
    "requestId": "7f4e8a1b-2c3d-4e5f-a6b7-c8d9e0f1a2b3",
    "environment": "production",
    "host": "api-server-3"
}

The structured version gives you: the ability to query all webhooks from a specific provider, all events related to a specific campaign, all events for a specific user, all events across a specific host, all events carrying a specific requestId — without writing custom parsers, without grep, and with sub-second query times in a properly configured log aggregation system.

If I were starting a system today, this is the first thing I'd lock in: every log entry is a structured JSON record. Everything downstream depends on it.

Setting This Up in NestJS with Pino

I switched from Winston to Pino on Helpa's backend after a load test at 400 concurrent requests revealed that Winston's synchronous transport was adding around 3–4ms to p95 request latency — not fatal, but visible and entirely attributable to logging overhead. The same test with Pino showed sub-millisecond logging cost. Pino serialises asynchronously and buffers writes to avoid blocking the event loop. At low traffic the difference is irrelevant. At the concurrency levels a viral campaign produces, a 3–4ms logging tax compounds across every request in the burst.

// logger.module.ts
import { Module } from "@nestjs/common";
import { ConfigService } from "@nestjs/config";
import { LoggerModule as PinoLoggerModule } from "nestjs-pino";

@Module({
    imports: [
        PinoLoggerModule.forRootAsync({
            inject: [ConfigService],
            useFactory: (config: ConfigService) => ({
                pinoHttp: {
                    level:
                        config.get("NODE_ENV") === "production"
                            ? "info"
                            : "debug",
                    transport:
                        config.get("NODE_ENV") !== "production"
                            ? {
                                  target: "pino-pretty",
                                  options: { colorize: true }
                              }
                            : undefined,
                    serializers: {
                        req(req) {
                            return {
                                method: req.method,
                                url: req.url
                            };
                        }
                    },
                    customProps: (req) => ({
                        service: config.get("SERVICE_NAME") ?? "api",
                        environment: config.get("NODE_ENV"),
                        requestId: req.headers["x-request-id"]
                    }),
                    redact: {
                        paths: [
                            "req.headers.authorization",
                            'req.headers["x-api-key"]',
                            "body.cardNumber",
                            "body.cvv",
                            "body.bvn",
                            "body.password"
                        ],
                        censor: "[REDACTED]"
                    }
                }
            })
        })
    ]
})
export class LoggerModule {}

The redact configuration is not optional. Every field in the path list is automatically replaced with [REDACTED] before the log is serialised. This means Authorization headers, API keys, BVNs, card numbers, and passwords can never appear in log output regardless of which developer writes what. It's enforced at the library level, not left to discipline. One caveat: the body.* paths only match when log calls nest the DTO under a body key. A call like this.logger.log({ cardNumber: dto.cardNumber }) passes fields at the root level and bypasses every body.* path in the list. The safe convention is to always pass domain data as { body: dto }, or add explicit root-level keys — cardNumber, bvn, etc. — to the redact list as well. I cover the broader security implications in Section 8.

The difference between pino-pretty in development and raw JSON in production is deliberate. In development, pretty-printed logs are dramatically easier to read. In production, you want raw JSON because your log aggregation pipeline — Logstash, Promtail, Fluentd — can parse it without brittle pattern matching. Pretty-printing in production just adds overhead and makes parsing harder.

The Standard Log Shape

Across both systems, we standardised on a log shape that every event must carry:

interface LogEntry {
    level: "debug" | "info" | "warn" | "error" | "fatal";
    timestamp: string; // ISO 8601
    service: string; // Which service emitted this
    event: string; // Namespaced event identifier: 'payment.webhook.received'
    requestId?: string; // HTTP and queue-processor events; absent on cron-initiated events which use runId instead
    userId?: string; // When the request is user-scoped
    tenantId?: string; // For multi-tenant systems like Festease
    environment?: string; // Injected by customProps on HTTP requests; absent on async/cron events not handled by nestjs-pino middleware
    host?: string; // Injected by customProps on HTTP requests; absent on async/cron events not handled by nestjs-pino middleware
    duration?: number; // Milliseconds, for operation timing
    message?: string; // Human-readable summary — passed as the second argument to logger calls: logger.log(mergeObject, 'message')
    [key: string]: unknown; // Domain-specific context (reference, amount, etc.)
}

The event field uses dot-notation namespacing intentionally. payment.webhook.received, payment.webhook.processed, payment.webhook.failed, donation.settled, donation.reconciliation.triggered. This creates a taxonomy of system events that makes query construction intuitive: show me everything under payment.webhook.* across the last hour.

3. Correlation IDs: Tracking a Request Across Its Entire Lifecycle

The single most impactful change I made to incident response capability was introducing correlation IDs — a unique identifier assigned to each request at the boundary of the system and propagated through every downstream operation that request generates.

Without a correlation ID, a request that touches multiple services, two database calls, a Redis cache lookup, and a webhook queue job produces log entries distributed across all of those components with no shared field that ties them together. You have timestamps, and you can try to reconstruct the sequence by time, but if two requests are in flight simultaneously — which is always the case in production — you are manually separating interleaved log streams from memory.

With a correlation ID, every log entry for everything a single request caused carries the same requestId. In Kibana or Grafana Loki, one query returns the complete story.

The Implementation

The correlation ID is generated (or forwarded if one arrives in the request header) at the middleware layer:

// correlation-id.middleware.ts
import { Injectable, NestMiddleware } from "@nestjs/common";
import { Request, Response, NextFunction } from "express";
import { v4 as uuidv4 } from "uuid";

@Injectable()
export class CorrelationIdMiddleware implements NestMiddleware {
    use(req: Request, res: Response, next: NextFunction): void {
        const incomingId = req.headers["x-request-id"] as string;
        const requestId = incomingId ?? uuidv4();

        // Attach to request so downstream services can read it
        req["requestId"] = requestId;
        req.headers["x-request-id"] = requestId;

        // Expose on response so the caller can correlate their own logs
        res.setHeader("x-request-id", requestId);

        next();
    }
}

Why accept an incoming x-request-id rather than always generating a new one? Because your clients and upstream services have their own logs. When a Paystack webhook fires and you want to trace that flow from Paystack's retry logs through your ingestion endpoint through your queue processor, the same requestId needs to appear in all of them. If your gateway or proxy generates the ID before it reaches your service, you preserve that chain. If you generate a fresh one on arrival, you break it.

Propagating Through Async Queues

The part that most implementations get wrong is propagation through asynchronous boundaries. HTTP middleware propagates the ID through the synchronous request lifecycle. But once you enqueue a job — a webhook to process, a settlement to confirm, a reconciliation to run — that job executes in a different process, often much later. If the requestId isn't embedded in the job payload, it's gone.

On Helpa, every queue job includes a meta object:

interface JobMeta {
  requestId: string;
  correlatedAt: string;   // When the job was enqueued
  source: string;         // What enqueued it: 'webhook-handler', 'cron', 'api'
}

// When enqueuing
const job = await this.donationQueue.add('process-webhook', {
  payload: webhookData,
  meta: {
    requestId: req['requestId'],
    correlatedAt: new Date().toISOString(),
    source: 'webhook-handler',
  },
});

this.logger.log({
  event: 'webhook.job.enqueued',
  requestId: req['requestId'],
  reference: webhookData.reference,
  jobId: job.id,
});

// In the queue processor
@Process('process-webhook')
async handleWebhook(job: Job<WebhookJobData>): Promise<void> {
  const { payload, meta } = job.data;

  this.logger.log({
    event: 'webhook.processing.started',
    requestId: meta.requestId,
    provider: payload.provider,
    reference: payload.reference,
    attempt: job.attemptsMade,
  });

  // ... processing logic
}

When that webhook processor later throws an exception, the error log carries the same requestId as the HTTP log entry from when the webhook was first received. One query in your log aggregation system retrieves the complete chain. Even when a processor exits silently without throwing — as in the opening incident — that shared requestId across the two log entries that were emitted (webhook.received and webhook.job.enqueued) is what made the gap visible as a correlated pattern rather than two unrelated events.

Note that Paystack's reference is not the same as your requestId. The reference identifies a specific transaction on Paystack's side — it lets you correlate a webhook back to the original charge. Your requestId is an internal identifier that tracks a request across your own services. Where the two meet is in the webhook handler: logging both together means you can move in either direction — from your internal request trace outward to Paystack's records, or from a Paystack reference inward to your processing logs.

Propagating to External Service Calls

Any outbound HTTP call to an external service should forward the requestId as x-request-id. This creates traceability across system boundaries:

// http-client.service.ts
async post<T>(url: string, body: unknown, requestId: string): Promise<T> {
  const response = await this.httpService.axiosRef.post<T>(url, body, {
    headers: {
      'x-request-id': requestId,
      'Content-Type': 'application/json',
    },
  });
  return response.data;
}

This applies to any downstream service that honours the x-request-id header — internal microservices, third-party APIs, or your own async workers invoked over HTTP.

4. Log Levels: Strategy, Not Convention

Every logging framework ships with log levels. Most teams treat the choice of level as a rough approximation — a stylistic preference. It isn't. Log levels are the primary mechanism for controlling signal-to-noise ratio in production, and miscalibrating them is a failure mode I've seen quietly neutralise an entire on-call rotation's ability to respond to real incidents.

The levels and what they actually mean in production:

Level	When to use it	Who reads it
`DEBUG`	Detailed execution context useful during development or targeted investigation	Engineers, disabled by default in production
`INFO`	Significant business events that should always be recorded: request received, donation settled, webhook processed	Dashboards, alerting, post-incident review
`WARN`	Something unexpected happened but the system recovered: a retry succeeded, a cache miss forced a database fallback, a third-party API returned a non-standard response	On-call alerts with low-priority routing
`ERROR`	An operation failed and the system could not recover it automatically: a webhook could not be processed after exhausting retries, a settlement failed	PagerDuty / immediate on-call routing
`FATAL`	The service is unable to continue: database connection lost at startup, configuration missing	Immediate escalation, service restart required

The failure I see most often is treating ERROR as a synonym for "something went wrong that I want to notice." That produces a system where every handled exception logs at ERROR, and after a few months of growth, your ERROR log stream is dominated by recoverable edge cases — validation failures, 404s, expired tokens — and the genuine ERROR-level events (failed settlements, lost webhook jobs) are invisible in the noise.

The mental model that actually works: an ERROR log means a human needs to investigate and take action. If the system can handle it automatically — retry, fallback, return a validation response to the client — it isn't an error from the system's perspective, even if something went wrong from the user's.

The Noise Problem and How We Fixed It

At a fintech I worked at, our error dashboards were triggering alerts dozens of times a day. The on-call rotation was becoming desensitised — engineers were starting to dismiss alerts reflexively because most of them turned out to be nothing. This is one of the most dangerous failure modes in production ops: alert fatigue causes real incidents to be dismissed alongside the noise.

When I audited the error log volume, roughly 60% of ERROR entries were user-triggered validation failures: invalid transfer amounts, expired payment sessions, duplicate transaction attempts. These are completely expected events in a payments system. They should be WARN at most. The remaining 40% were genuine system errors.

After reclassifying log levels based on recovery semantics, the error alert volume dropped by 65%. The on-call rotation started treating error alerts as real again.

@Post('transfer')
async initiateTransfer(
  @Body() dto: InitiateTransferDto,
  @Req() req: Request,
): Promise<TransferResponse> {
  try {
    const result = await this.transferService.initiate(dto, req['requestId']);

    this.logger.log({
      event: 'transfer.initiated',
      requestId: req['requestId'],
      userId: dto.userId,
      accountId: dto.accountId,
      amount: dto.amount,
      provider: dto.provider,
    });

    return result;
  } catch (error) {
    if (error instanceof TransferValidationException) {
      // User error — not a system failure
      this.logger.warn({
        event: 'transfer.validation.failed',
        requestId: req['requestId'],
        reason: error.message,
        userId: dto.userId,
      });
      throw error;
    }

    // Genuine system failure — requires investigation
    this.logger.error({
      event: 'transfer.initiation.failed',
      requestId: req['requestId'],
      userId: dto.userId,
      accountId: dto.accountId,
      error: error.message,
      stack: error.stack,
    });
    throw error;
  }
}

The four pillars covered here — structured log records, a standard log shape, correlation IDs propagated through every async boundary, and log levels enforced as contracts rather than convention — are the foundation of a queryable, incident-ready observability system.

The operational impact of building this properly is measurable. Before structured logging and correlation IDs, the incident described in this article's opening took over two hours to diagnose — manually cross-referencing dashboards and raw log files with no shared identifier across the output. After introducing the practices in this article, a structured log query returned the complete request chain in under a minute. Adding centralised log aggregation — without proactive alerting yet — brought mean time to detection down to around 18 minutes for a comparable incident: the information was queryable, but an engineer still had to notice it. With structural log-based alerting layered on top, that dropped to 8 minutes. With metrics-based alerting, 3 minutes. The 85% reduction in the subtitle isn't a projected estimate — it's the gap between a two-hour reconstruction and a three-minute alert.

Designing a Reliable Payment API for Real-Time Donations

Temitope Bamidele — Mon, 06 Apr 2026 22:31:35 +0000

A Retrospective on Moving Money at Scale — What I Built, What I Learned, and How It Evolved

Introduction

Helpa is a donation platform that connects donors to campaigns for medical bills and emergency relief across Nigeria — serving both individual fundraisers and NGOs. Its payment infrastructure settles real money for real emergencies: a delayed settlement means someone's surgery fundraiser stalls; a duplicate charge means a donor loses trust in the platform.

As the architect and technical lead, I designed and built the system from its first transaction through to a production-hardened platform serving multiple payment providers. The result: a 99.7% settlement success rate across three payment providers, no confirmed duplicate charges since deploying idempotency and atomic webhook processing, and automated recovery that has saved hundreds of donations from manual intervention.

This article documents the architecture, the failure modes discovered and resolved, and the engineering trade-offs involved.

Role and Context

Career context: I have spent over seven years building systems and payment infrastructure across the Nigerian fintech and banking sectors — from early-stage startups to consumer credit platforms, to banks processing billions daily. Before Helpa, I was a founding engineer at a buy-now-pay-later platform, where I built the checkout SDK, merchant integration layer, and multi-lender orchestration pipeline that processed consumer credit decisions across multiple banking partners with incompatible APIs and compliance requirements.

Role at Helpa: Architect and technical lead. I designed the original system, made every architectural decision documented here, and implemented the core reliability improvements described.

Team: I lead the payment infrastructure — setting the architectural direction, establishing engineering standards, and building the core systems, with implementation support from the broader engineering team.

System scope: The platform handles five distinct payment paths — frontend-initiated, webhook-confirmed (async), webhook-before-frontend (race condition), virtual account / offline payments, and recurring subscriptions — across three payment providers (Paystack, Flutterwave, Stripe), each with incompatible APIs, webhook formats, amount units, and settlement behaviours. The engineering complexity comes from the combinatorial surface: 5 payment paths × 3 providers × 2 entity types (individual and organisation) × multi-party settlement splits — with every combination requiring correct fee calculation, idempotent processing, and auditable fund movement.

Duration: Over four years operating this system in production, spanning its evolution from an MVP with a single payment provider to a hardened multi-provider platform — drawing on over seven years of backend and payment systems experience.

Influence beyond Helpa: The engineering standards established here — the prioritised remediation framework (reconciliation first, webhook hardening second, idempotency third, state machine migration fourth, circuit breakers fifth), the provider normaliser pattern, and the webhook security standards — are now the default approach for all payment-related development at Helpa.

1. Problem Context

What Makes Donations Different from Generic Payments

A donation system sits at an uncomfortable intersection of constraints that generic payment integrations don't face. The architecture documented here was designed to address each of these systematically:

Multiple entity types share the same payment rails. In our case, campaigns can be created by individual users or by organisations. This means separate wallet models, separate transaction models, separate donor records, and nearly every payment flow forks into two paths. The theoretical literature on payment systems assumes a single merchant model. Reality produces processUserSettlement and processNgoSettlement — two parallel methods that do almost the same thing but can't be trivially unified because the domain objects differ in ways that matter for accounting. The duplication is a deliberate trade-off: premature unification carries regression risk when the underlying domain models diverge on fee structures, settlement rules, and audit requirements.

Donations aren't a single amount. A donor gives ₦5,000 to a campaign and tips ₦500 to the platform. The payment gateway charges fees on the combined ₦5,500. The gateway fee has to be proportionally split across the donation and the tip. Our original implementation did this with floating-point arithmetic in major units:

// ❌ Original implementation — floating-point arithmetic on naira values
const percentageOfAmountTipped = (amountTipped / processorAmount) * 100;
const amountTippedSettled =
    (processorAmountSettled / 100) * percentageOfAmountTipped;
const amountDonatedSettled = processorAmountSettled - amountTippedSettled;

Get this wrong and the books stop balancing. Get it subtly wrong and totals may look correct in aggregate while individual transactions are still incorrect. We saw both failure modes during accounting audits. The subtler variant — aggregate-correct but transaction-wrong — drove the move to the integer minor-unit approach in Section 7's splitSettlement, which removed rounding drift entirely.

Payment providers don't agree on what "amount" means. Paystack returns amounts in kobo (divide by 100). Flutterwave returns naira. Stripe returns in the currency's smallest unit. Paystack includes fees in the response; Flutterwave returns amountSettled; Stripe requires a separate balanceTransactions.retrieve call. Every normalisation layer ends up with provider-specific branches, and every new provider breaks the abstraction. Currency unit mismatches across providers are a known source of catastrophic errors — e.g., crediting 100x the actual donation amount due to a kobo/naira confusion. This class of risk directly motivated the normaliser pattern, which enforces unit conversion at the provider boundary before any business logic executes.

Not every payment starts at the API. Some payment flows — such as bank transfers to dedicated virtual accounts — arrive exclusively via provider notifications with no prior API call. There's no pre-created transaction record. The webhook handler has to look up the wallet, create the credit transaction retroactively, and process the entire donation from scratch. The system must accommodate flows that start mid-lifecycle, which led to the EXTERNALLY_INITIATED state in the state machine design (Section 3).

Traffic is driven by emotion, not calendars. Unlike e-commerce where teams can prepare for Black Friday, donation spikes are unannounced. We've seen spikes of thousands of concurrent donations from a single viral social media post — 50x normal traffic in minutes. This informed the connection pooling and timeout strategies in production — every request holding a database connection for the duration of a payment gateway call is a connection pool exhaustion vector under spike load.

The Failure Modes This Architecture Defends Against

Failure Mode	Root Cause	Defence Mechanism
Duplicate donations	Client retries on timeout, webhook processed twice	Idempotency layer (Section 5)
Zombie transactions	Payment stuck in pending state because webhook was lost	Reconciliation cron (Section 9)
Ghost charges	Payment succeeds at gateway but webhook handler threw an unhandled exception	Reconciliation cron (Section 9); transactional outbox designed as target architecture (Section 6)
Split-brain state	Webhook arrives before frontend verification completes	State machine with atomic transitions (Section 3)
Phantom success	DB records success but gateway initialisation actually failed	Provider verification + state preconditions

Each of these failure modes informed a specific architectural decision. The sections that follow explain the defence in depth.

2. Production Architecture

The Target Architecture (Deployed Incrementally)

The target production architecture comprises a load balancer, API gateway (rate limiting, auth, request signing), a state machine engine, and isolated message queues per provider — with a transactional outbox relay designed for deployment when throughput exceeds the reconciliation layer's coverage window (Section 6). Currently, the reconciliation cron serves as the active compensating control for the dual-write gap. What matters isn't the topology — it's the reasoning behind each component, and the ROI of each decision.

Migration Priority (What I Shipped and When)

As I resolved each failure class in production, I developed a prioritised remediation framework that is now the standard for all payment infrastructure work across Helpa. Here's the sequence, ordered by ROI:

Reconciliation cron — catches zombie transactions. Highest ROI, lowest complexity. Prevents the worst class of payment failures with minimal surface area. (Implemented — runs every 6 hours, auto-recovers stale transactions)
Webhook error handling — always return 200, process async. Eliminates an entire class of duplicate processing at the ingestion layer. (Implemented across all providers)
Idempotency keys on donation initiation — prevents duplicate charges from the client side. (Implemented — unique index + Redis cache for fast-path dedup)
State machine replacing boolean flags — prevents impossible states, reduces incident debugging time from hours to minutes. (Implemented — state enum and transition engine deployed; expand-contract migration underway to retire legacy boolean flags)
Transactional outbox — eliminates dual-write risk between DB and queue. (Designed and ready for deployment; reconciliation cron is the active compensating control at current throughput — see Section 6)
Circuit breakers — prevents gateway outage cascades from propagating through the queue system. (Implemented with calibrated thresholds derived from three months of production gateway behaviour data — Flutterwave P99 latency, error rate baselines, and recovery time patterns)

Five of the six items are fully deployed; the transactional outbox (item 5) is designed and ready to deploy when the next throughput threshold is reached. The prioritisation order itself has become a standard I use when evaluating payment infrastructure maturity — reconciliation first, always.

3. The Donation Lifecycle — All Five Paths

What the Simple Version Looks Like

Client → POST /donate → Generate reference → Save Transaction → Return payment link → Client pays → Webhook → Verify → Settle

This is the textbook flow. In production, it is incomplete for real-world payment behaviour.

What Actually Happens

There are at least five distinct paths a payment can take through the system. The descriptions below use the original boolean-flag notation; the state machine design later in this section replaces these flags with explicit state transitions:

Path 1: Frontend-initiated (happy path)
Client calls initializePayment → Transaction record created with isVerified: false → Client pays via gateway → Client calls verifyClientPayment → Gateway verification → Transaction updated to isVerified: true → Donor record created → Ledger entry recorded → Wallet credited

Path 2: Webhook-confirmed (async)
Same as Path 1, but the webhook queue job finishes fully before the client's browser redirect fires the frontend verification call — a sequential outcome, not a concurrent one. The isVerifiedByWebhook: true flag short-circuits the frontend call idempotently when it arrives.

Path 3: Webhook-before-frontend (race condition)
The webhook and the frontend verification call arrive nearly simultaneously — both executing concurrently. Some providers dispatch webhooks within milliseconds of payment completion, faster than the client's browser can redirect. Without state preconditions, both the webhook processor and the frontend verify call would attempt to settle the same transaction at the same time. The state machine's atomic findOneAndUpdate with a state precondition ensures only one concurrent path wins the update; the other finds no matching document and returns idempotently. This is the scenario the direct INITIALIZED → SETTLED transition is designed for.

Path 4: Virtual account / offline payment
A bank transfer arrives with no prior API call. No transaction record exists. The webhook handler creates it retroactively with isVerified: true and processes the entire donation flow from scratch.

Path 5: Recurring subscription
Paystack sends a webhook for a recurring charge. The system finds the subscription plan, locates (or creates) the recurring donor record, processes the payment, and appends the transaction ID to the donor's subscriptionTransactions array. Each renewal is a new transaction but not a new donor record.

Any state machine or lifecycle model that doesn't account for all five paths is incomplete.

Why Boolean Flags Were the Root Cause

The original schema used:

isVerified — has the payment been confirmed?
isVerifiedByWebhook — was it confirmed by a webhook?
isVerifiedByFrontend — was it confirmed by frontend verification?
isFrontendInitiated — did the flow start from the frontend?
isForRecurrent — is this a recurring payment?
isAnonymous — anonymous donor?

Of these, the first four are lifecycle flags — they control payment state and create a space of 2^4 = 16 possible combinations, most of which are invalid (isVerified: false, isVerifiedByWebhook: true shouldn't exist but nothing in the schema prevents it). The remaining two (isForRecurrent, isAnonymous) are domain properties that persist regardless of state — a recurring donation is still recurring whether it's pending or settled. The state machine replaces the four lifecycle booleans; the domain booleans remain as independent fields.

With boolean flags, answering "what state is this transaction in?" requires mentally decoding a combination of flags across multiple service files. This cognitive load — and the invalid state combinations it enables — is why a formal state machine replaces the lifecycle flags.

The State Machine Design

enum DonationState {
    CREATED = "CREATED", // Transaction saved, not yet paid
    INITIALIZED = "INITIALIZED", // Payment link generated, awaiting payment
    PROCESSING = "PROCESSING", // Gateway confirms attempt, verifying
    SETTLED = "SETTLED", // Funds confirmed and recorded
    FAILED = "FAILED", // Terminal: payment failed
    EXPIRED = "EXPIRED", // Terminal: payment link TTL exceeded
    REFUNDED = "REFUNDED", // Terminal: funds returned to donor
    EXTERNALLY_INITIATED = "EXTERNALLY_INITIATED" // Virtual account / offline
}

const VALID_TRANSITIONS: Record<DonationState, DonationState[]> = {
    [DonationState.CREATED]: [DonationState.INITIALIZED, DonationState.EXPIRED],
    [DonationState.INITIALIZED]: [
        DonationState.PROCESSING,
        DonationState.EXPIRED,
        DonationState.SETTLED
    ],
    [DonationState.PROCESSING]: [DonationState.SETTLED, DonationState.FAILED],
    [DonationState.SETTLED]: [DonationState.REFUNDED],
    [DonationState.FAILED]: [], // Terminal state
    [DonationState.EXPIRED]: [], // Terminal state
    [DonationState.REFUNDED]: [], // Terminal state
    [DonationState.EXTERNALLY_INITIATED]: [
        DonationState.SETTLED,
        DonationState.FAILED
    ]
};

Note INITIALIZED → SETTLED as a valid transition. This handles the webhook-before-frontend race (Path 3). A textbook state machine would only allow INITIALIZED → PROCESSING → SETTLED, but in practice, Paystack's webhook can beat the frontend verification call. The direct INITIALIZED → SETTLED path accommodates this without rejecting valid settlements.

The Transition Engine

The transition engine uses MongoDB's findOneAndUpdate with a state precondition — the condition and the update execute as a single atomic operation:

const updated = await this.transactionModel.findOneAndUpdate(
    { _id: donationId, state: currentExpectedState },
    { $set: { state: targetState }, $push: { stateHistory: transitionRecord } },
    { new: true }
);

if (!updated) {
    // State already changed — concurrent transition won
    // This is idempotent by design
    return;
}

The findOneAndUpdate with a state precondition achieves atomic transitions in MongoDB — the condition and the update are a single operation, so no concurrent webhook can slip through.

State history as an immutable audit log. The stateHistory array provides a complete timeline of every state change, who triggered it, and when. When a donor disputes a charge, the team can reconstruct exactly what happened without manually querying raw database records. This pattern is now the enforced standard across all payment-related code reviews.

Migrating from Booleans to States

The migration follows a standard expand-contract pattern:

Add a state field alongside the existing booleans
Backfill it with a script that maps flag combinations to states (e.g., isVerified: true → SETTLED, isVerified: false, created > 2h ago → EXPIRED)
Write code that sets both the new state field and the old boolean flags in the same operation
Migrate read paths gradually — one query at a time
Once all reads use state, drop the boolean fields

This is a standard expand-contract migration with low risk — both representations coexist during the transition, and the rollback path is simply reverting reads to the boolean fields.

4. API Design — Defensive by Default

The Donation Endpoint (Production Design)

@Post('donate')
@UseGuards(RateLimitGuard)
@UsePipes(new ValidationPipe({ whitelist: true, forbidNonWhitelisted: true }))
async donate(
  @Body() dto: CreateDonationDto,
  @Headers('Idempotency-Key') idempotencyKey: string,
) {
  if (!idempotencyKey) {
    throw new BadRequestException('Idempotency-Key header is required');
  }

  // Acquire a distributed lock to prevent concurrent requests
  // with the same idempotency key from racing
  const lock = await this.lockService.acquire(
    `donation:idempotency:${idempotencyKey}`,
    { ttl: 30_000 },
  );

  try {
    // Check if we've already processed this idempotency key
    const cached = await this.idempotencyStore.get(idempotencyKey);
    if (cached) {
      return cached.response; // Return the original response, byte-for-byte
    }

    const donation = await this.donationService.create(dto, idempotencyKey);

    const paymentLink = await this.paymentService.initializePayment({
      amount: donation.amountMinor,
      currency: donation.currency,
      reference: donation.reference,
      callbackUrl: this.config.get('PAYMENT_CALLBACK_URL'),
      metadata: {
        donationId: donation.id,
        campaignId: donation.campaignId,
        idempotencyKey,
      },
    });

    // Transition state: CREATED → INITIALIZED
    await this.donationService.transition(
      donation.id,
      DonationState.INITIALIZED,
      { paymentLink },
    );

    const response = {
      donationId: donation.id,
      reference: donation.reference,
      paymentLink,
      expiresAt: new Date(Date.now() + 30 * 60 * 1000), // 30min TTL
    };

    // Cache response for idempotency (TTL: 24 hours)
    await this.idempotencyStore.set(idempotencyKey, { response }, 86_400);

    return response;
  } finally {
    await lock.release();
  }
}

Design Decisions and Why

Idempotency-Key as a first-class citizen. The client generates a UUID before making the request. If the network drops the response, the client retries with the same key and gets back the exact same response. This isn't optional — it's the only way to safely handle mobile clients on unreliable networks. On mobile-dominant platforms, the most common duplicate trigger is users tapping the donate button twice on a slow connection. Idempotency keys are the primary defence against this.

Distributed lock before the idempotency check. Without this, two concurrent requests with the same idempotency key can both pass the get() check before either writes. The lock serialises them. We use a short TTL (30 seconds) so a crashed process doesn't hold the lock forever.

Explicit state transition. We don't update a state field directly. The transition() method validates that the transition is legal (CREATED → INITIALIZED is valid; FAILED → INITIALIZED is not) and writes the state change with a state history record atomically via findOneAndUpdate. When the outbox pattern is deployed (Section 6), this method will additionally write an outbox event in the same transaction.

Expiry on payment links. Payment links that live forever are a security and UX liability. A 30-minute TTL with a background job that moves expired INITIALIZED donations to EXPIRED prevents zombie transactions from accumulating in the system.

5. Idempotency — Architecture and Implementation

Webhook Processing — Atomic State Transitions

The core principle: collapse the read and write into a single atomic operation. A read-then-conditional-write pattern (separate findOne followed by findOneAndUpdate) leaves a race window between the two calls — during webhook retry storms, concurrent requests can slip through that gap. The atomic approach eliminates this entirely:

const updated = await this.transactionModel.findOneAndUpdate(
    {
        transactionReference,
        state: DonationState.INITIALIZED,
        paymentProcessor: "paystack"
    },
    {
        $set: {
            state: DonationState.SETTLED,
            transactionDetails: verifiedData
            // ... all settlement fields
        },
        $push: { stateHistory: transitionRecord }
    },
    { new: true }
);

if (!updated) {
    // Either doesn't exist or already settled — both are idempotent outcomes
    this.logger.debug(
        `No unverified transaction found for ${transactionReference} — likely already processed`
    );
    return;
}

// Enqueue downstream processing (donor record, wallet, notification)
await this.downstreamQueue.add("payment.confirmed", {
    reference: transactionReference,
    amountMinor: updated.amountMinor,
    paidAt: new Date(),
    provider: updated.paymentProcessor,
    webhookEventId
});

This is a single atomic operation. No race window. The state: DonationState.INITIALIZED precondition guarantees that only one concurrent request "wins" the update. During the expand-contract migration (Section 3), both the state field and legacy boolean flags are set in the same operation — reads migrate to state while writes maintain backward compatibility until the booleans are retired.

Donation Initiation — Database-Level Safety Net

The distributed lock and Redis cache shown in Section 4 handle the primary deduplication path. The database provides a secondary safety net via a unique index on the idempotency key:

// In the schema
idempotencyKey: {
  type: String,
  unique: true,
  sparse: true  // allows null for legacy records
}

If two concurrent requests with the same idempotency key bypass the distributed lock (e.g., during a Redis failover), the unique index catches the duplicate at the database level. The donation creation flow (Section 4) is wrapped in a try/catch — if MongoDB throws an E11000 duplicate key error, the handler returns the cached original response:

catch (err) {
    if (err.code === 11000) {
        // Duplicate key on idempotencyKey — return the cached original response
        const cached = await this.idempotencyStore.get(idempotencyKey);
        if (cached) return cached.response;

        // Redis cache expired — reconstruct from DB
        const existing = await this.transactionModel.findOne({ idempotencyKey });
        return {
            donationId: existing._id,
            reference: existing.transactionReference,
            paymentLink: existing.paymentLink,
            expiresAt: existing.expiresAt
        };
    }
    throw err;
}

The distributed lock (Section 4) is the primary defence. The unique index is the database-level safety net — belt-and-braces. Together, they guarantee that even under infrastructure failures, no duplicate donation is created.

The idempotency key is separate from the payment reference because they serve different purposes: the reference is the system's internal identifier for the payment gateway; the idempotency key is the client's identifier for "this intent to donate."

6. The Dual-Write Problem — And Why It's Hiding in Our Codebase

Here's what our webhook handler does after verifying a payment:

// 1. Update Transaction in MongoDB
await this.transactionModel.findOneAndUpdate(/* ... */);

// 2. Trigger downstream processing via Bull queue
this.processDownstreamQueue(data);

If step 1 succeeds and step 2 fails (Redis is down, queue is full, process crashes between the two lines), the payment is marked as verified but the campaign wallet is never credited, the donor record is never created, and the ledger entry is never written. The donor sees "success" but the campaign never receives the funds.

This is the classic dual-write problem — writing to two systems (MongoDB and Redis/Bull) without a shared transaction boundary. The two lines of code look harmless. Identifying this risk led to my design of the transactional outbox pattern for this system.

The Transactional Outbox Solution

The following outbox architecture is designed and ready for deployment when throughput exceeds the reconciliation layer's coverage window. Currently, the reconciliation cron (Section 9) serves as the active compensating control for the dual-write gap. The design is documented here as the target architecture.

Instead of publishing to the queue directly, the outbox pattern writes an outbox document in the same MongoDB transaction as the state change:

const session = await this.connection.startSession();
session.startTransaction();

try {
    await this.transactionModel.findOneAndUpdate(
        { transactionReference, state: DonationState.INITIALIZED },
        {
            $set: { state: DonationState.SETTLED },
            $push: { stateHistory: transitionRecord } /* ... */
        },
        { session, new: true }
    );

    await this.outboxModel.create(
        [
            {
                aggregateType: "Transaction",
                aggregateId: transactionReference,
                eventType: "PaymentVerified",
                payload: data,
                publishedAt: null
            }
        ],
        { session }
    );

    await session.commitTransaction();
} catch (err) {
    await session.abortTransaction();
    throw err;
} finally {
    session.endSession();
}

A separate polling process reads unpublished outbox events and adds them to the Bull queue:

@Cron('*/5 * * * * *')  // Every 5 seconds
async drainOutbox() {
  const events = await this.outboxModel
    .find({ publishedAt: null })
    .sort({ _id: 1 })
    .limit(100);

  for (const event of events) {
    try {
      await this.queue.add(event.eventType, event.payload, {
        attempts: 5,
        backoff: { type: 'exponential', delay: 2000 }
      });

      await this.outboxModel.updateOne(
        { _id: event._id },
        { publishedAt: new Date() }
      );
    } catch (err) {
      this.logger.error(`Outbox drain failed for ${event._id}`, err);
      break;  // Preserve ordering — don't skip ahead
    }
  }
}

Why break on failure? For a given aggregate, events should drain in order. A donation can't be settled before it's verified. If event #3 fails to publish, skipping to event #4 risks violating causal ordering. The trade-off: a single failed publish blocks the entire batch, including events for unrelated campaigns. At higher throughput, this drain would be partitioned by aggregate (e.g., per campaign), so a failure in one partition doesn't block others. At current scale, the global drain with break is simple and sufficient — the 5-second polling interval bounds the delay.

Important: This requires MongoDB replica sets for multi-document transactions. On a standalone MongoDB instance, transactions aren't available — the alternative is the findOneAndUpdate atomic operation with the reconciliation layer (Section 9) as the active compensating control for the residual dual-write risk, which is our current production configuration.

Outbox Hygiene

A practical concern with the transactional outbox: the collection grows forever unless it's pruned. A daily cron deletes successfully published events older than seven days. Unpublished events are kept indefinitely — they're evidence of failures that need investigation.

Architectural Trade-off

The outbox pattern is the theoretically complete solution, but engineering is about trade-offs under real constraints. At our current throughput, the reconciliation layer (Section 9) provides equivalent coverage — it catches the occasional missed queue publish within its 6-hour cycle, well within our SLA. Bull/Redis has demonstrated consistent reliability across our production workloads.

I shipped the reconciliation layer as the production-grade dual-write mitigation, designed the outbox architecture for the next throughput tier, and identified the throughput threshold (when reconciliation cycle time exceeds our settlement SLA) that would trigger deployment. This is a deliberate architectural layering — the reconciliation cron is the shipped solution at current scale, not a temporary workaround.

For higher throughput, Debezium (CDC via database change streams) eliminates the polling delay entirely. It's the right tool at scale, but it introduces a dedicated change-data-capture pipeline to operate and monitor.

7. Multi-Provider Payment Normalisation

How Response Formats Actually Differ

// Paystack: amounts in kobo, fees in response
processorAmount = result.data.data.amount / 100;
processorAmountSettled =
    result.data.data.amount / 100 - result.data.data.fees / 100;
transactionReference =
    result.data.data.channel === "dedicated_nuban"
        ? result.data.data.customer.customer_code
        : result.data.data.reference;

// Flutterwave: amounts in naira, settled amount in response
processorAmount = result.data.amount;
processorAmountSettled = result.data.amount_settled;
transactionReference = result.data.tx_ref;

// Stripe: amounts in smallest currency unit, need separate API call for fees
const paymentIntent = await stripe.paymentIntents.retrieve(
    payload.data.object.payment_intent
);
const balanceTransaction = await stripe.balanceTransactions.retrieve(
    paymentIntent.charges.data[0].balance_transaction
);
processorAmount = balanceTransaction.amount / 100;
processorAmountSettled =
    balanceTransaction.amount / 100 - balanceTransaction.fee / 100;

Three providers. Three amount formats. Three fee calculation methods. Three ways to extract the transaction reference. And I haven't shown the currency-specific edge cases or the authorisation/card-object shapes that differ. This code is spread across our webhook handlers — each handler has its own version of this normalisation logic. A bug in one doesn't get fixed in the others.

The Normaliser Pattern

I learned early not to pursue a universal payment interface. Instead, I normalise at the boundary:

interface NormalisedPaymentResult {
    provider: "paystack" | "flutterwave" | "stripe";
    transactionReference: string;
    amountMinor: number; // Always in minor units (kobo, cents) as integers
    settledAmountMinor: number; // After gateway fees, in minor units
    currency: string;
    rawResponse: any; // Keep the original for debugging
    providerTransactionId: string;
}

class PaystackNormaliser {
    normalise(raw: any): NormalisedPaymentResult {
        return {
            provider: "paystack",
            transactionReference:
                raw.data.channel === "dedicated_nuban"
                    ? raw.data.customer.customer_code
                    : raw.data.reference,
            amountMinor: raw.data.amount, // Paystack already returns kobo
            settledAmountMinor: raw.data.amount - raw.data.fees,
            currency: raw.data.currency,
            rawResponse: raw.data,
            providerTransactionId: String(raw.data.id)
        };
    }
}

class FlutterwaveNormaliser {
    normalise(raw: any): NormalisedPaymentResult {
        return {
            provider: "flutterwave",
            transactionReference: raw.data.tx_ref,
            amountMinor: Math.round(raw.data.amount * 100), // Flutterwave returns naira → convert to kobo
            settledAmountMinor: Math.round(raw.data.amount_settled * 100),
            currency: raw.data.currency,
            rawResponse: raw.data,
            providerTransactionId: String(raw.data.id)
        };
    }
}

Each provider gets its own normaliser class. The webhook handler calls the appropriate normaliser and works exclusively with NormalisedPaymentResult from that point on. Adding a new provider means writing one normaliser class — not a new 300-line branch in every webhook handler. When we added the third provider, the integration took a fraction of the time the earlier ones had.

The Split Amount Problem

After normalisation, the settled amount still needs to be split proportionally across donation + tip:

// All amounts are in minor units (kobo/cents) as integers
function splitSettlement(
    donationAmountMinor: number,
    tipAmountMinor: number,
    totalSettledMinor: number
): { donationSettledMinor: number; tipSettledMinor: number } {
    const totalMinor = donationAmountMinor + tipAmountMinor;
    if (totalMinor === 0)
        return { donationSettledMinor: 0, tipSettledMinor: 0 };

    const donationSettledMinor = Math.round(
        (donationAmountMinor / totalMinor) * totalSettledMinor
    );
    const tipSettledMinor = totalSettledMinor - donationSettledMinor; // NOT computed from ratio

    return { donationSettledMinor, tipSettledMinor };
}

Note: tipSettled = totalSettled - donationSettled rather than computing both from ratios. This complementary calculation ensures the two values always sum exactly to totalSettled, avoiding the rounding drift that independent ratio calculations produce. I identified this class of bug during a quarterly accounting audit — the original implementation computed both values from ratios, which balanced in aggregate but not per transaction. For maximum precision, all intermediate calculations use amounts in minor units (kobo/cents) as integers.

Currency Conversion Timing

Our system applies conversion_rate_to_naira at donation creation time, not at settlement time. This is a deliberate decision with trade-offs:

Approach	Advantage	Disadvantage
Lock rate at intent	Donor and campaign amounts are deterministic	Requires accurate rate capture at initiation
Apply rate at settlement	Reflects settlement-time market rate	Campaign impact varies with timing

We chose intent-time conversion to build trust with donors and campaigners through deterministic outcomes. The donor sees the amount in their source currency, and the naira value credited is fixed from initiation.

I documented this as an architectural decision record (ADR) after re-deriving the reasoning myself — future teammates will inevitably wonder why the conversion rate differs between what the donor saw and what the campaign received. Having to reconstruct undocumented reasoning from first principles is itself a clear signal that undocumented decisions create engineering overhead.

8. Webhook Handling — Architecture and Security Model

Architectural Decisions That Proved Critical

Saving raw webhooks before processing. Every webhook gets saved to a provider-specific collection (webhook_stripe, webhook_flutterwave, webhook_paystack) before any business logic runs. When processing fails, the raw payload is preserved, and webhooks can be replayed or processed via one-off scripts. This decouples ingestion from processing and guarantees no payment data is lost regardless of downstream failures.

Queue-based verification. After saving the webhook, the actual verification runs in a Bull queue with retries:

async enqueueWebhookVerification(provider: string, webhookData: any): Promise<void> {
  const queue = this.providerQueues[provider]; // e.g., 'paystack-webhook', 'flutterwave-webhook'
  queue.add(
    webhookData,
    {
      removeOnComplete: true,
      attempts: 5,
      backoff: { type: 'exponential', delay: 2000 },
    }
  );
}

Each provider's queue is a separate Bull instance (this.providerQueues maps provider names to queue instances), ensuring true fault isolation. This decouples webhook receipt from processing, which means the webhook endpoint responds fast.

Independent verification. We don't trust webhook payloads. After a webhook arrives, we call the gateway's verification API (stripe.paymentIntents.retrieve, Flutterwave's /transactions/:id/verify, Paystack's /transaction/verify/:reference) to confirm the payment independently. The webhook is a notification; the verification API is the source of truth.

Separate Bull queues per provider. If Paystack is having a bad day, its queue backs up while Flutterwave and Stripe processing continues unaffected. Each provider gets its own queue instance — not just a named processor on a shared queue — so a saturated Paystack queue cannot block Flutterwave or Stripe workers:

// Each provider registers its own Bull queue instance
@Processor("paystack-webhook")
export class PaystackWebhookProcessor {
    @Process()
    async handle(job: Job) {
        /* ... */
    }
}
// FlutterwaveWebhookProcessor and StripeWebhookProcessor follow the same pattern
// with their own queue names ('flutterwave-webhook', 'stripe-webhook')

Vulnerabilities Identified and Remediated

Webhook exception handling policy.

// ❌ This causes the gateway to retry, which causes duplicate processing
if (payload.type !== "checkout.session.completed") {
    throw new HttpException(
        "Unsupported event type",
        HttpStatus.UNPROCESSABLE_ENTITY
    );
}

When a webhook handler throws, the gateway receives a non-200 response and retries — the same webhook can be processed more than once. To prevent this class of failure, webhook handlers were designed from the outset to return 200 and process anomalies asynchronously:

// ✅ Always return 200 — log anomalies, don't throw
if (payload.type !== "checkout.session.completed") {
    this.logger.debug(`Ignoring Stripe event type: ${payload.type}`);
    return { status: "ok" };
}

Leaking endpoint existence on invalid signatures.

// ❌ Flutterwave: tells attackers this is a real webhook endpoint
if (payload !== config.flutterwaveWebhookSecret) {
    throw new HttpException("Invalid Header", HttpStatus.UNAUTHORIZED);
}

// ❌ Paystack: uses !== instead of timing-safe comparison
if (hash !== headers["x-paystack-signature"]) {
    throw new HttpException("Invalid Header", HttpStatus.UNAUTHORIZED);
}

Both problems: returning non-200 responses confirms the endpoint exists and invites probing. The Paystack signature check uses !==, which short-circuits on the first differing character — leaking information about the expected signature via timing side-channels.

// ✅ Timing-safe comparison + always return 200
import { timingSafeEqual, createHmac } from "crypto";

function verifyPaystackSignature(
    rawBody: Buffer,
    signatureHeader: string
): boolean {
    const hash = createHmac("sha512", config.paystackSecretKey)
        .update(rawBody)
        .digest("hex");

    try {
        return timingSafeEqual(Buffer.from(hash), Buffer.from(signatureHeader));
    } catch {
        return false; // Buffers of different lengths
    }
}

// In the handler — use the raw request body, not the parsed object:
if (
    !verifyPaystackSignature(request.rawBody, headers["x-paystack-signature"])
) {
    this.logger.warn("Invalid Paystack webhook signature", { ip: request.ip });
    return { status: "ok" }; // Don't throw, don't leak
}

The Stripe verification vulnerability. A security review of the webhook infrastructure revealed that Stripe signature verification failures were not enforcing a hard reject in the handler path — meaning forged Stripe webhooks could be processed. This was a critical vulnerability: verification failures were swallowed and execution continued. The root cause: the verification used the API key instead of the webhook secret, the body was pre-parsed JSON instead of the required raw Buffer, and failure control flow did not terminate processing. The fix:

function verifyStripeSignature(
    rawBody: Buffer,
    signatureHeader: string
): boolean {
    try {
        stripe.webhooks.constructEvent(
            rawBody,
            signatureHeader,
            config.stripeWebhookSecret // WEBHOOK secret, not API key
        );
        return true;
    } catch {
        return false;
    }
}

Note: Stripe webhook verification requires the raw request body (Buffer), not the parsed JSON. NestJS's default body parser pre-parses the body, so a raw body middleware was added specifically for the webhook route. This is the most common reason Stripe webhook verification fails in NestJS/Express applications — and the source of a bug caught during the initial Stripe integration.

The General Principle for Webhook Endpoints

Every webhook handler across providers follows this sequence — codified as a team checklist and enforced through both code review and the ESLint rule described in Section 15:

Verify signature using the raw request body — configure raw body middleware for webhook routes (NestJS/Express parse bodies by default); reject silently (return 200, log anomaly)
Deduplicate — check webhook log by event ID
Log payload BEFORE processing (mask sensitive fields — card numbers, emails, and account data) — enables replay if processing fails
Verify independently — call the gateway's verification API; never trust webhook data alone
Publish to queue for async processing — decouple receipt from business logic

9. The Reconciliation Layer — The Foundation of Payment Reliability

No matter how robust the webhook handling, idempotency, and error handling are, state will drift. Webhooks will be lost. Processing will fail in unanticipated ways. Gateway retries will be exhausted.

The reconciliation job is a core reliability control. It is the safety net for every other pattern in this article and the highest-ROI investment in the infrastructure. Across hundreds of recovery events, the reconciliation layer has consistently recovered payment drift that escaped earlier controls.

@Cron('0 */6 * * *')  // Every 6 hours
async reconcileStaleTransactions() {
  const runId = `run_${uuidv4().split('-')[0]}`;
  const twoHoursAgo = new Date(Date.now() - 2 * 60 * 60 * 1000);

  // Expire transactions that never got a payment link
  await this.transactionModel.updateMany(
    { state: DonationState.CREATED, created: { $lt: twoHoursAgo } },
    { $set: { state: DonationState.EXPIRED }, $push: { stateHistory: { from: 'CREATED', to: 'EXPIRED', source: 'reconciliation', at: new Date() } } }
  );

  // Find transactions stuck in non-terminal state (these have gateway records to verify)
  const stale = await this.transactionModel.find({
    state: { $in: [DonationState.INITIALIZED, DonationState.PROCESSING] },
    created: { $lt: twoHoursAgo }
  }).limit(200);

  this.logger.log({
    event: 'reconciliation.run.found_stale',
    runId,
    count: stale.length,
  });

  for (const tx of stale) {
    try {
      let gatewayStatus: NormalisedPaymentResult | null = null;

      if (tx.paymentProcessor === 'paystack') {
        const client = new PaystackClient(this.logger);
        const result = await client.verifyPayment({
          reference: tx.transactionReference
        });
        if (result.data?.status === 'success') {
          gatewayStatus = new PaystackNormaliser().normalise(result);
        }
      }

      if (tx.paymentProcessor === 'flutterwave') {
        const client = new FlutterwaveClient(this.logger);
        const result = await client.verifyPayment({
          transactionId: tx.processorTransactionId
        });
        if (result.data?.status === 'successful') {
          gatewayStatus = new FlutterwaveNormaliser().normalise(result);
        }
      }

      if (tx.paymentProcessor === 'stripe') {
        const paymentIntent = await stripe.paymentIntents.retrieve(
          tx.processorTransactionId
        );
        if (paymentIntent.status === 'succeeded') {
          const balanceTransaction = await stripe.balanceTransactions.retrieve(
            paymentIntent.charges.data[0].balance_transaction
          );
          gatewayStatus = new StripeNormaliser().normalise(balanceTransaction);
        }
      }

      // Transition explicitly terminal gateway statuses to FAILED
      // Only fail on conclusive statuses — 'pending' or 'processing' at the gateway
      // means the payment may still complete (e.g., slow bank transfers)
      if (!gatewayStatus && this.isTerminalGatewayFailure(tx)) {
        await this.transactionModel.findOneAndUpdate(
          { _id: tx._id, state: { $in: [DonationState.INITIALIZED, DonationState.PROCESSING] } },
          {
            $set: { state: DonationState.FAILED },
            $push: { stateHistory: { from: tx.state, to: 'FAILED', source: 'reconciliation', at: new Date() } }
          }
        );
        continue;
      }

      if (gatewayStatus) {
        this.logger.warn({
          event: 'reconciliation.settlement.missing',
          runId,
          reference: tx.transactionReference,
          provider: tx.paymentProcessor,
        });

        // Enqueue through the same payment.confirmed queue as webhooks,
        // so downstream processing goes through Section 10's idempotency guard
        await this.downstreamQueue.add('payment.confirmed', {
          reference: tx.transactionReference,
          amountMinor: gatewayStatus.amountMinor,
          paidAt: new Date(),
          provider: tx.paymentProcessor,
          webhookEventId: `reconciliation:${tx.transactionReference}`,
        });
      }

      // Rate limit: don't hammer the gateway API
      await new Promise(resolve => setTimeout(resolve, 500));
    } catch (err) {
      this.logger.error({
        event: 'reconciliation.transaction.failed',
        runId,
        reference: tx.transactionReference,
        error: err.message,
      });
      continue;  // Don't let one failure stop the batch
    }
  }

  this.metrics.gauge('reconciliation.stale_donations', stale.length);
}

What This Catches

Webhooks that were never delivered (gateway outage during retry window)
Webhooks that were delivered but whose processing threw an unhandled exception
Webhooks that were delivered but whose downstream queue job failed all retries
Virtual account payments where wallet lookup failed due to a temporary DB issue
The dual-write gap — Transaction updated but Bull queue publish failed

Operational Notes

Rate limit gateway API calls. A reconciliation job that fires 200 verify requests simultaneously will get rate-limited or banned. The 500ms delay between calls is deliberate.
Skip transactions that are too fresh. Two hours is our threshold. Some gateways take minutes to settle. Reconciling a transaction that's 10 minutes old might catch it mid-processing.
Log with a source: 'reconciliation' tag to distinguish reconciled settlements from normal ones in the audit trail.
Monitor the count of reconciled transactions. If reconciliation is fixing more than a handful of transactions per run, something upstream is broken. That's a signal, not a solution.

10. Event-Driven Processing — Handling the Downstream

Sections 5 and 8 together cover the webhook processing path — Section 5 for the atomic state transition, Section 8 for the ingestion and security layer. This section covers the downstream processing that follows: creating donor records, crediting wallets, writing ledger entries, and sending notifications. The webhook handler in Section 5 enqueues a payment.confirmed event after the atomic update succeeds; the queue consumer below picks it up.

@Process('payment.confirmed')
async handlePaymentConfirmed(job: Job<PaymentConfirmedEvent>) {
  const { reference, amountMinor, paidAt, provider, webhookEventId } = job.data;

  const donation = await this.donationService.findByReference(reference);

  if (!donation) {
    // Webhook for a donation we never created — this is a reconciliation issue
    this.logger.error('Orphaned payment webhook', { reference, provider });
    await this.alertService.trigger('orphaned-payment', { reference, amountMinor });
    return; // Don't throw — this job shouldn't retry
  }

  // Verify amount matches to prevent amount manipulation attacks
  // donation.amountMinor and the event's amountMinor are both in minor units (kobo/cents)
  if (donation.amountMinor !== amountMinor) {
    this.logger.error('Amount mismatch', {
      expected: donation.amountMinor,
      received: amountMinor,
      reference,
    });
    await this.alertService.trigger('amount-mismatch', { reference, donation, amountMinor });
    return;
  }

  // Idempotency guard: if downstream processing already completed for this event,
  // skip to avoid duplicate wallet credits, donor records, and ledger entries on retries
  const alreadyProcessed = await this.webhookLogRepo.findOne({ eventId: webhookEventId, processedAt: { $ne: null } });
  if (alreadyProcessed) {
    this.logger.debug(`Event ${webhookEventId} already processed — skipping`);
    return;
  }

  // Guard: if the donation isn't settled yet (e.g., reconciliation enqueued this event
  // for a payment that was confirmed at the gateway but missed webhook processing), transition it now
  if (donation.state !== DonationState.SETTLED) {
    try {
      await this.donationService.transition(donation.id, DonationState.SETTLED, {
        paidAt,
        provider,
        webhookEventId,
      });
    } catch (err) {
      if (!(err instanceof ConflictException)) {
        throw err; // Unknown error — let the queue retry
      }
      // ConflictException means another process settled it — safe to continue
    }
  }

  // Downstream processing: donor record, wallet credit, ledger, notifications
  await this.settlementService.processDownstream(donation, { amountMinor, paidAt, provider });

  // Mark webhook log as processed
  await this.webhookLogRepo.update(
    { eventId: webhookEventId },
    { processedAt: new Date() },
  );
}

Retry Strategy

The queue uses the same exponential backoff configuration shown in Section 8 (5 attempts, 2-second base delay). Completed jobs are retained for 24 hours for debugging; failed jobs stay in the dead-letter queue indefinitely — I've diagnosed production issues by reading DLQ job payloads.

11. Resilience Patterns — Calibrated for Reality

Circuit Breaker — For Payment Gateway Calls

Without circuit breakers, a struggling gateway cascades failures through the webhook queue — every retry hits the same broken endpoint.

import CircuitBreaker from "opossum";

// Calibration based on our actual gateway behaviour:
// - Flutterwave P99 latency is ~3s, so 10s timeout is generous
// - We typically do 50-100 verify calls/hour, so volumeThreshold of 10
//   means we need 10 failures before the circuit considers opening
// - 30s reset gives the provider time to recover from transient issues
//   without leaving donations unprocessed for too long

const breakerOptions = {
    timeout: 10_000,
    errorThresholdPercentage: 50,
    resetTimeout: 30_000,
    volumeThreshold: 10
};

// One circuit breaker per provider — consistent with per-provider queue isolation
this.providerBreakers = {
    paystack: new CircuitBreaker(
        (params) => this.verifyWithPaystack(params),
        breakerOptions
    ),
    flutterwave: new CircuitBreaker(
        (params) => this.verifyWithFlutterwave(params),
        breakerOptions
    ),
    stripe: new CircuitBreaker(
        (params) => this.verifyWithStripe(params),
        breakerOptions
    )
};

Each breaker emits open, halfOpen, and close events — we log state transitions and alert on-call when a circuit opens, since that means donations for that provider are queuing up.

Calibration matters more than the pattern. An errorThresholdPercentage of 50% with a volumeThreshold of 3 will open the circuit after 2 failures out of 3 calls — far too sensitive. We chose 10 because it means we need sustained failures, not a blip. I arrived at these numbers by looking at our actual error logs over three months.

What happens when the circuit is open? Jobs queue up in Bull and retry after the resetTimeout. If the gateway recovers, the circuit closes and the backlog drains. If it doesn't, jobs eventually hit their max retry count and land in the dead-letter queue for manual investigation. The reconciliation cron picks up anything the DLQ misses.

Concurrency Control — The Campaign Goal Race Condition

if (isCampaignValid.amountRaised >= isCampaignValid.amountToRaise) {
    throw new HttpException(
        "Campaign amount goal has been met",
        HttpStatus.BAD_REQUEST
    );
}

This check has a race condition. Two concurrent donations can both pass this check (both see amountRaised < amountToRaise) and both succeed — pushing the campaign over its goal. For Helpa, this is acceptable: raising more than the goal isn't catastrophic. But if business rules require a hard cap, the solution is an atomic check-and-increment:

const result = await this.campaignModel.findOneAndUpdate(
    {
        _id: campaignId,
        isBanned: false,
        isClosed: false,
        isPublished: true,
        $expr: { $lt: ["$amountRaised", "$amountToRaise"] }
    },
    { $inc: { amountRaised: donationAmount } },
    { new: true }
);

if (!result) {
    throw new HttpException(
        "Campaign goal has been met or campaign is unavailable",
        HttpStatus.BAD_REQUEST
    );
}

This is atomic — the condition and the increment happen in one database operation. No race window. But it changes the flow: amountRaised is incremented optimistically and must be decremented if downstream processing fails. I chose to accept the race condition since exceeding a campaign goal isn't catastrophic in our domain.

Rate Limiting — Layered

Our current rate limiting is global (100 requests/minute per IP), with webhook endpoints excluded from IP-based rate limiting since legitimate provider traffic during spike events could exceed these thresholds. This is a reasonable starting point but insufficient for a donation platform. The next layer requires per-user limits (a real user donates once, not 10 times per minute), per-campaign limits (preventing a single viral campaign from starving others), and webhook-specific limits keyed to provider identity rather than IP (preventing endpoint flooding while allowing legitimate traffic). These are the additions that would make the rate limiting model complete.

12. Multi-Entity Settlements and Domain Complexity

Two domain complexities distinguish this system from textbook payment integrations: entity-type duality and multi-party settlement routing.

The organisation/user duality. Campaigns can be created by individual users or organisations, producing parallel flows: Wallet vs OrgWallet, Donor vs OrgDonor, Transaction vs OrgTransaction. The same fee calculation bug surfaced in both paths during separate audits — confirming that code duplication across entity types compounds into a reliability risk. The shipped solution: extract shared business logic (settlement splitting, fee calculation, amount normalisation) into pure functions that both flows call. The orchestration layers remain separate, but the critical calculations are shared and tested once. This eliminated the cross-path bug class entirely.

Affiliate commission routing. When a campaign has an affiliate marketer, every donation splits three ways — campaign wallet, platform revenue, and affiliate wallet. The affiliate commission is calculated on the original donation amount, not the settled amount (after gateway fees), meaning the total disbursed (campaign + platform + affiliate) exceeds processorAmountSettled. The platform absorbs the difference — effectively paying the affiliate's proportional share of gateway fees. This is a structural cost of the commission model, not a rounding artifact. The settlement pipeline handles this three-way split atomically.

13. Logging

Structured logging through every operation, with a consistent schema across all processors and crons. Every log entry carries a namespaced event identifier and a correlation ID — requestId for queue jobs (propagated from the originating HTTP request via JobMeta), runId for cron-initiated runs. This makes it possible to query the complete trail for a single donation — from webhook receipt through signature verification, queue enqueue, processor execution, and settlement — in a single Kibana filter. Getting this schema locked in early matters: retrofitting it onto a running system means rebuilding every existing query, dashboard, and log triage habit against the new field names.

The standard across both webhook processing and reconciliation:

this.logger.log({
  event: 'webhook.signature.verified',
  requestId: reqId,
  user: sanitizeForLog(payload),  // userId + masked email via log-sanitiser.ts
  reference: payload.reference,
  provider: payload.provider,
});

Every processor and cron emits a namespaced event identifier and a correlation ID — requestId for queue jobs (carrying the value from the originating HTTP request), runId for cron runs. This makes it possible to query the complete trail for a single donation — from webhook receipt through signature verification, queue enqueue, processor execution, and settlement — in a single Kibana filter.

14. Security — Proactive Hardening of Webhook Infrastructure

A comprehensive security audit of the webhook infrastructure across all payment providers established the following security controls:

Threat Vector	Control Implemented	Priority
Timing side-channel on signature	`crypto.timingSafeEqual()` for all provider signature verification	High
Webhook signature spoofing	Provider-specific verification using webhook secrets + raw body parsing	Critical
Endpoint existence leakage	Consistent 200 responses for all webhook endpoints, anomalies logged silently	Medium
Webhook endpoint flooding	IP allowlisting using provider-published IP ranges	Low
Retry storms from non-200 responses	All webhook handlers return 200, process asynchronously via queue	Critical

Security controls in payment infrastructure are treated as architectural requirements, not feature work — they are prioritised ahead of product development.

Defence in Depth Principles

Validate webhook signatures using timing-safe comparison — crypto.timingSafeEqual(), not ===. String equality short-circuits on the first different character, leaking information about the expected signature via timing side-channels.
Never log full payment details. Mask card numbers, account numbers, and personally identifiable information. Log references and IDs, not raw payloads.
Apply webhook-specific rate controls. Webhook endpoints are excluded from generic IP-based rate limiting (Section 11), but they still need protection — IP allowlisting using provider-published ranges (above) is the first layer, and provider-keyed rate limits prevent endpoint flooding while allowing legitimate traffic. An attacker who discovers a webhook URL can flood it with forged payloads; even though signatures are verified, the verification itself consumes compute.
Audit trail — every state change, every API call, every admin action goes into an append-only audit log. In financial systems, this isn't optional. It's also how you survive regulatory audits without losing sleep.

15. Architectural Principles for Payment Infrastructure

These are the engineering principles established from designing and operating Helpa's payment infrastructure:

1. Shared business logic across entity types. Extract critical calculations — settlement splitting, fee calculation, amount normalisation — into pure, tested functions that all entity-type flows call. Orchestration layers can remain separate when domain models genuinely diverge, but duplicating business logic across Transaction and OrgTransaction means every bug is fixed twice (or isn't). For greenfield systems, prefer a single collection with a discriminator field from day one.

3. Provider normaliser layer from day one. All gateway-specific logic behind normaliser classes that output a standard NormalisedPaymentResult. The webhook handler doesn't know what Paystack is. This pattern prevents an entire class of amount-format mismatches and ensures new providers integrate cleanly.

4. Reconciliation as a first-class architectural requirement. A reconciliation layer is the highest-ROI component in a payment system. Helpa's reconciliation cron runs every 6 hours, automatically recovering zombie transactions and verifying payment state against gateway APIs. It addresses core drift classes — lost webhooks, failed queue jobs, and dual-write gaps. It should be the first reliability investment in any payment system.

5. Idempotency key on the initiation endpoint. Unique index on the key field. Redis cache for fast-path dedup. This prevents the most common user-visible bug (double charges) with minimal complexity.

6. Never throw in webhook handlers. This is enforced through an ESLint custom rule that flags throw statements inside @Process()-decorated methods within classes matching *WebhookProcessor — codifying the principle into automated tooling rather than relying on code review.

7. Own the subscription state. Don't let a payment provider own the subscription lifecycle. The system should own the subscription state, reconcile against the gateway on each cycle, and be able to continue processing renewals even if the provider changes — provider portability requires owning the state.

16. Trade-offs — Key Decisions and Rationale

Decision	Benefit	Cost	Rationale
Transactional Outbox	No dual-write risk	Outbox collection, relay process, needs replica set	Staged rollout — reconciliation currently mitigates this risk within SLA at present throughput
State Machine	Impossible states unrepresentable	Migration from boolean flags, more upfront design	Implemented — the migration cost only grows with time
Reconciliation Cron	Catches core drift classes	Gateway API rate limits, monitoring surface	First thing I built — highest ROI in our operating context
Provider Normalisers	Single processing path	Abstraction maintenance, new provider onboarding	Essential with 3 providers — eliminated per-provider bug classes
Idempotency Keys	No duplicate charges	Redis for caching, unique index on keys	Non-negotiable for payment endpoints
Circuit Breakers	Graceful degradation	Calibration effort, false opens during normal variance	Calibrated from three months of production gateway behaviour data
Atomic State Precondition (`findOneAndUpdate`)	No race conditions on state transitions	Failed transitions require retry or idempotent handling	Required for concurrent webhook/frontend state transitions
Unified Data Model	Single bug-fix path, simpler queries	Major migration, risk of regression	Extracted shared calculations — full unification scoped for next phase
Separate Queue Pools per Provider	Fault isolation	More infrastructure, more queue configuration	Implemented — a Paystack queue backup no longer blocks Stripe processing
Webhook Signature + Independent Verification	Defence in depth	Extra API call per webhook, added latency	Both are non-negotiable for payment security

Conclusion

Measurable Impact

The architecture documented in this article delivers:

99.7% settlement success rate across three payment providers with incompatible APIs, webhook formats, and settlement behaviours — each requiring its own normaliser, signature verification, and reconciliation path
No confirmed duplicate charges since idempotency keys and atomic webhook processing were deployed — across a system where any single donation can arrive via five different payment flows and fork into two entity-type paths
Automated recovery via the reconciliation layer — hundreds of donations auto-recovered across major incidents that would otherwise have required manual intervention
System resilience under viral traffic — sustained thousands of concurrent donation attempts during campaign spikes without duplicate charges, missed settlements, or degraded processing
Timing-safe webhook security across all providers — signature verification, raw body handling, and consistent 200-response patterns
Per-transaction settlement accuracy — floating-point rounding errors in fee splitting eliminated through the complementary calculation pattern
Provider fault isolation — a Paystack degradation no longer cascades to Flutterwave and Stripe processing

Engineering Principles Established

The patterns in this article — transactional outbox, state machines, idempotency, reconciliation — draw from years of financial systems engineering. The contribution here is applying them pragmatically in production environments where teams often optimise for short-term delivery under pressure. The engineering challenge is knowing when simplicity is sufficient and when it becomes a liability — and that judgment is codified into the prioritised remediation framework and architectural standards that govern how the team builds payment infrastructure. The stack is implementation detail; the principles are what determine reliability.

The core principle: know which shortcuts compound and which don't. Missing reconciliation compounds. Code duplication across entity types compounds. These are areas where targeted engineering effort produced compounding reliability gains — each fix addressed an entire class of failure, not just a single defect.

The prioritised remediation framework — reconciliation first, webhook hardening second, idempotency third, state machine migration fourth, circuit breakers fifth — reflects years of building and operating payment systems and understanding which improvements deliver disproportionate reliability gains. The transactional outbox completes the sequence when throughput demands it.

Designing a Scalable Multi-Tenant Architecture with NestJS: Lessons from Building Real-World Systems

Temitope Bamidele — Mon, 30 Mar 2026 22:43:22 +0000

I've spent the last few years building multi-tenant platforms, most recently Festease (an event management platform), on NestJS with a mix of MongoDB and PostgreSQL. Along the way, I got a lot of things wrong before I got them right, and this post is my attempt at sharing what actually held up in production versus what sounded good in a design doc.

If you've ever shipped a multi-tenant system and found yourself debugging why Tenant A's data showed up in Tenant B's dashboard at 2am, this one's for you.

Why multi-tenancy in the first place?

If you've built or used any SaaS product, you've almost certainly encountered multi-tenancy, whether you realized it or not. Slack, Shopify, Notion, most of the tools we use daily are multi-tenant under the hood. It's basically the default architecture for SaaS at this point.

The pitch for multi-tenancy is simple: you want one codebase, one deployment, and logical separation between clients. The alternative is spinning up a separate instance for every customer, which works fine until you have 15 of them and a migration to run.

Every tenant on our platforms needed their own data, their own configurations, sometimes their own branding. But nobody wanted to pay for dedicated infrastructure, at least not at first. So the question became: how do you give each customer the feeling of having their own system without actually running one for each of them?

That's multi-tenancy. And the tricky part isn't getting it to work. It's getting it to work safely, at scale, without drowning in operational overhead.

The three ways to slice it

There are really three patterns, and you've probably seen them before:

Shared database, shared schema is the simplest. Everyone lives in the same tables, and a tenantId column keeps things separate. It's cheap, easy to migrate, and it works great until someone writes a query without a WHERE tenantId = ? clause and suddenly you've got a data leak. We used this on Festease. For a platform that's early and moving fast, this is the right call.

Shared database, separate schema is the middle ground. Each tenant gets their own PostgreSQL schema, like tenant_abc.orders or tenant_xyz.orders. The database engine enforces the boundary, which is reassuring. The downside is that every schema migration has to run N times, once per tenant, and if one fails halfway through you're dealing with schema drift. Good tooling might help, but it's still more work.

Separate database per tenant is maximum isolation. Each customer gets their own database instance. You'd think this solves everything, but it creates a different set of problems: connection pool management becomes a headache, backups multiply, and monitoring gets noisy etc. This is better for large enterprises with enterprise clients.

Here's how the three compare visually:

Here's what I actually recommend: don't pick one. Use them as tiers. Start most tenants on shared schema because it's the fastest to build and cheapest to operate. When a client asks for exclusivity or their traffic starts impacting others, move them up. When enterprise clients needed data isolation guarantees, migrate them to dedicated databases. Building the tenant resolution layer as an abstraction over the data source early enough would mean the application code won't change at all when clients come knocking for exclusivity.

How we wired it up in NestJS

The implementation boils down to three things: figuring out which tenant a request belongs to, attaching that context to the request, and making sure every database query respects it.

Before diving into the code, it helps to understand where each piece sits in NestJS's request pipeline:

Middleware runs first, so by the time guards, interceptors, or controllers touch the request, the tenant context is already there.

For tenant resolution, we wrote a simple middleware that pulls the tenant ID from the x-tenant-id header and validates it:

// multi-tenant.middleware.ts
@Injectable()
export class MultiTenantMiddleware implements NestMiddleware {
    private readonly TENANT_ID_PATTERN = /^[a-zA-Z0-9_-]{3,64}$/;

    use(req: Request, res: Response, next: NextFunction) {
        const tenantId = req.headers["x-tenant-id"] as string;

        if (!tenantId) {
            throw new BadRequestException("Tenant ID is required");
        }

        if (!this.TENANT_ID_PATTERN.test(tenantId)) {
            throw new BadRequestException("Invalid tenant ID format");
        }

        req["tenantId"] = tenantId;
        next();
    }
}

That regex validation is doing more work than it looks like. The tenant ID ends up in database queries, Redis cache keys, log entries, all of it. If you let someone pass in a tenantId with special characters or absurd length, you're opening yourself up to injection attacks or just weird cache pollution. Validate at the boundary.

We register it globally in AppModule:

@Module({
    /* ... */
})
export class AppModule implements NestModule {
    configure(consumer: MiddlewareConsumer) {
        consumer.apply(MultiTenantMiddleware).forRoutes("*");
    }
}

Then a decorator to pull the tenant ID cleanly in controllers:

// tenant.decorator.ts
export const Tenant = createParamDecorator(
    (data: unknown, ctx: ExecutionContext): string => {
        const request = ctx.switchToHttp().getRequest();
        return request["tenantId"];
    }
);

One decision that paid off: the decorator reads from request['tenantId'] rather than directly from headers. The middleware is the only thing that touches the raw header. This means we can swap the tenant resolution strategy later. We may derive it from a subdomain, or from the JWT itself. Nothing downstream has to change.

Making sure every query is scoped

On the data access side, tenant scoping looks like this:

@Injectable()
export class OrdersService {
    constructor(
        @InjectModel(Order.name) private orderModel: Model<OrderDocument>
    ) {}

    async findAll(tenantId: string): Promise<Order[]> {
        return this.orderModel.find({ tenantId }).exec();
    }

    async create(tenantId: string, dto: CreateOrderDto): Promise<Order> {
        return this.orderModel.create({ ...dto, tenantId });
    }
}

One rule we enforced strictly: the tenantId never comes from the request body. It always comes from the middleware or the authenticated user's JWT. If you let the client tell you which tenant they belong to via the request body, you've given them the ability to impersonate other tenants.

Defense in depth with a guard

Even with the middleware in place, we added a guard that cross-checks the tenant ID from the header against the one in the JWT:

@Injectable()
export class TenantGuard implements CanActivate {
    canActivate(ctx: ExecutionContext): boolean {
        const request = ctx.switchToHttp().getRequest();
        const headerTenantId = request["tenantId"];
        const jwtTenantId = request.user?.tenantId;

        if (jwtTenantId && headerTenantId !== jwtTenantId) {
            throw new ForbiddenException("Tenant mismatch: access denied");
        }

        return true;
    }
}

Is this belt-and-suspenders? Yes. But I've seen enough incidents caused by a new developer adding a route without realizing the middleware wasn't covering it. The guard is cheap insurance.

Authentication and who gets to do what

Every JWT we issue includes the tenant ID:

async login(user: ValidatedUser): Promise<{ accessToken: string }> {
  const payload = {
    sub: user.id,
    tenantId: user.tenantId,
    roles: user.roles,
  };

  return {
    accessToken: this.jwtService.sign(payload, { expiresIn: '15m' }),
  };
}

For authorization, we went with roles scoped to each tenant rather than global roles. A user might be an admin on one tenant and a regular member on another. The active tenant is determined per-request, not cached in a session. Early on we made the mistake of caching the tenant context in the user's session, and users who belonged to multiple organizations would switch tenants in the UI but keep seeing data from the previous one until they logged out and back in.

Three role levels worked for us: tenant:admin for full control within a tenant, tenant:member for regular users, and platform:admin for our internal team to manage tenants and system config.

Database-level details that matter

With MongoDB on the shared schema model, indexing becomes critical fast. A simple index on tenantId alone isn't enough once you're past a few thousand records per tenant. We added compound indexes:

orderSchema.index({ tenantId: 1, createdAt: -1 });
orderSchema.index({ tenantId: 1, status: 1 });

Without those, every "show me recent orders" query was doing a collection scan filtered by tenant. After adding them, p95 query time dropped from ~400ms to under 20ms.

On the PostgreSQL side with separate schemas, the setup is cleaner from an isolation perspective, but you have to be diligent about migrations:

CREATE SCHEMA IF NOT EXISTS tenant_abc;

CREATE TABLE tenant_abc.orders (
    id UUID PRIMARY KEY DEFAULT gen_random_uuid(),
    user_id UUID NOT NULL,
    items JSONB NOT NULL,
    status VARCHAR(20) DEFAULT 'pending',
    created_at TIMESTAMPTZ DEFAULT NOW()
);

For tenants on dedicated databases, you would need a connection resolver that lazily creates and caches connections:

@Injectable()
export class TenantConnectionService {
    private connections = new Map<string, Connection>();

    async getConnection(tenantId: string): Promise<Connection> {
        if (this.connections.has(tenantId)) {
            return this.connections.get(tenantId);
        }

        const config = await this.configService.getTenantDbConfig(tenantId);
        const connection = await createConnection(config);
        this.connections.set(tenantId, connection);

        return connection;
    }
}

Early on, we didn't have any eviction on this cache. Connections to inactive tenants just accumulated until the process started running out of file descriptors. We added an LRU (Least Recently Used) policy that closes connections idle for more than 10 minutes. Simple fix, embarrassing that it took a production incident to add it.

Scaling and performance

A few things that held up once you crossed 200+ active tenants and roughly 10K daily requests:

You need connection pooling. We used PgBouncer in front of PostgreSQL and Mongoose's built-in pooling for MongoDB. Shared-schema tenants all share one pool. Dedicated-DB tenants get isolated pools with hard connection limits. Without that, one tenant's bulk data import can starve everyone else.

For caching, we namespaced every Redis key with the tenant ID:

const cacheKey = `tenant:${tenantId}:orders:${orderId}`;
await this.redis.set(cacheKey, JSON.stringify(order), "EX", 300);

This sounds obvious, but early on we had a bug where two tenants with an order of the same ID were getting each other's cached data. Embarrassing, caught in QA, never happened again.

We also varied cache TTLs by tier. Premium tenants got longer TTLs, free-tier tenants got shorter ones. It helped keep Redis memory manageable without adding more instances.

Set up per-tenant rate limiting early. It's tempting to skip it when traffic is low, but all it takes is one tenant running a bulk data import or a misconfigured integration to saturate your shared resources. Give each tenant rate limits proportional to their tier so one tenant's spike doesn't become everyone's problem."

You can't fix what you can't see

The single biggest operational improvement we made was adding tenant-aware observability. Before that, when something broke, the first 15 minutes of every incident was spent figuring out which tenant was affected. Every log entry, every metric, every trace should carry the tenant ID.

We built a NestJS interceptor that attached tenant context to every log line:

@Injectable()
export class TenantLoggerInterceptor implements NestInterceptor {
    intercept(context: ExecutionContext, next: CallHandler): Observable<any> {
        const request = context.switchToHttp().getRequest();
        const tenantId = request["tenantId"];
        const start = Date.now();

        return next.handle().pipe(
            tap(() => {
                const duration = Date.now() - start;
                Logger.log({
                    tenantId,
                    method: request.method,
                    path: request.url,
                    duration,
                    status: context.switchToHttp().getResponse().statusCode
                });
            })
        );
    }
}

We tracked per-tenant request rates, error rates, and latency percentiles (p50, p95, p99). Eventually we built an internal dashboard that combined all of this into a per-tenant health score. It sounds like overkill, but it let us proactively reach out to tenants showing degradation before they opened a support ticket. That shift from reactive to proactive was a turning point for our ops maturity.

What I'd tell someone building this today

After doing this across multiple projects, a few things stand out:

Tenant context is a security boundary, not just a query filter. If a query runs without a tenantId, that's not a bug. It's a potential data breach. Enforce it everywhere: middleware, guards, repositories, cache keys, logs. If it's possible to forget, someone will.

Build the abstraction for tenant mobility early. The ability to move a tenant from shared schema to a dedicated database without changing application code is worth the upfront design cost. Retrofitting it later is painful and expensive.

NestJS's middleware and decorator patterns fit this problem really well. You can layer tenant resolution, validation, and scoping cleanly using the framework's built-in patterns. That said, NestJS isn't the only framework that supports this kind of layering. Express has middleware chains, Django has middleware classes and custom decorators, Spring Boot has filters and interceptors. The principles are the same; the wiring just looks different. Pick a framework that gives you clear request lifecycle hooks and lean into them.

Start with the cheapest isolation model, but measure everything from day one. Track per-tenant data volume, query times, and error rates. When the metrics tell you a tenant needs to move to a higher isolation tier, you'll have the data to justify the infrastructure cost, and you'll know exactly when to make the call.

Add rate limiting before you need it. You always think your tenants will behave predictably. They won't. One bulk import, one misconfigured webhook, one eager integration partner, and suddenly your shared resources are saturated. Rate limits are cheap. Incidents are not.

Invest in tenant-aware observability. This one change cut our mean time to resolution by more than half. When your logs and metrics are grouped by tenant, the first question in every incident ("who's affected?") has an instant answer. Everything moves faster from there.

Write down your architectural decisions. We started keeping Architecture Decision Records (ADRs) after the third time a new engineer asked "why do we do it this way?" Being able to point to a document that explains the trade-offs instead of reconstructing the context from memory made onboarding dramatically faster.

Wrapping up

There's no single right way to do multi-tenancy. The architecture that works depends on how many tenants you have, what their compliance requirements look like, and where you are in the product's life. What I've tried to lay out here are the patterns and trade-offs that held up for me and my team across real systems with real traffic.

The thing I keep coming back to is this: ship the simplest architecture that works today, but design the abstractions so they can handle what you'll need at 10x the scale. Don't over-engineer isolation for tenants who don't need it yet. Don't under-engineer it for tenants who do. Use the data to decide when to upgrade, and make sure you're collecting that data from the start.

That balance between simplicity now and flexibility later is the hardest part. It's also where the real engineering lives.