DEV Community

CloudveilElenor12
CloudveilElenor12

Posted on

Custom Logger Transport: HTTP Correlation IDs for Pipeline Incident Reconstruction

#nestjs #logging #observability

Short answer: use a custom NestJS logger transport to send structured logs to a backend API asynchronously, and make request_id plus trace_id mandatory at the boundary. For a nightly B2B SaaS data pipeline, this is enough to centralize evidence with little infrastructure work, but only if the event model is designed for incident reconstruction before anyone starts counting log volume.

The difficult choice is not HTTP versus an SDK. It is deciding which evidence will still matter at 03:10, when an account import has completed 48,991 rows and one tenant reports that the final nine are missing. Store too little and the incident cannot be reconstructed. Store every intermediate object and both cost and disclosure risk expand without a useful bound.

What should a NestJS custom logger send to an HTTP backend API?

Each event should carry timestamp, level, message, context, request_id, trace_id, and exception metadata. Those fields answer different questions: when the stage ran, how severe the outcome was, which component emitted it, which inbound operation initiated it, which distributed operation it belonged to, and what failed. A correlation ID must be propagated rather than generated anew inside every service. Otherwise the field looks complete but cannot join the pipeline's evidence.

For this workload, I would add stable business dimensions only where they support the investigation: service, pipeline_run_id, stage, and a tenant-safe identifier. I would not turn row IDs, email addresses, raw filenames, or exception messages into indexed labels. Prometheus documents the same underlying cardinality hazard for metrics: every distinct label combination creates another time series. Log indexes differ in implementation, but the budget question is identical — how many distinct values will this field create, and will an investigator actually filter on it?

Count first. Suppose a pipeline serves 2,000 tenants, has 12 stages, and uses 6 levels. Treating those three values as independently selectable dimensions already yields as many as 144,000 combinations before pipeline_run_id enters the index. The run ID is excellent correlation data and terrible low-cardinality metadata. Keep it searchable as a field, but don't casually promote it to a label in systems where labels define index streams.

One line can be useful.

curl --request POST \
  --url "${LOG_API_BASE}/v1/logs/ingest" \
  --header "Authorization: Bearer ${INFRAI_API_KEY}" \
  --header "Content-Type: application/json" \
  --header "Idempotency-Key: pipeline-run-20260816-stage-validate-summary" \
  --retry 4 \
  --retry-all-errors \
  --data '{
    "timestamp": "2026-08-16T19:00:42Z",
    "level": "error",
    "message": "Validation stage rejected records",
    "context": "NightlyImportWorker",
    "request_id": "req_01J5A8D2Q7",
    "trace_id": "4fd0b2c4e6134db9a46baf2f35d072cc",
    "exception": {
      "name": "ImportValidationError",
      "message": "9 records failed schema validation"
    },
    "service": "account-importer",
    "pipeline_run_id": "run_20260816_1900",
    "stage": "validate"
  }'
Enter fullscreen mode Exit fullscreen mode

LOG_API_BASE should be configured outside the source file, and the key must remain in a secret store. The explicit method makes the write unambiguous. The idempotency key protects the summary event if delivery is retried, while curl's retry behavior prevents a tight retry loop on HTTP 429. In application code, the transport should put events onto a bounded in-memory or durable queue and let a background worker batch delivery; request handling must not wait for the logging backend.

There is a catch. A bounded queue must have an explicit overflow policy. Dropping debug events may be rational; silently dropping the one terminal error is not. Reserve capacity for error and completion events, record a local counter for dropped lower-priority events, and flush with a short deadline during graceful shutdown. Don't promise lossless delivery unless a durable queue actually provides it.

Retention is part of the event schema

Retention math should precede vendor selection. Estimate daily stored bytes as events per run × average encoded bytes × runs per day × retention days, then add index overhead according to the system being evaluated. If 2,000 tenant runs each emit 40 events averaging 900 bytes, the raw payload is about 72 MB per night and roughly 2.16 GB over 30 nights, before replicas and indexes. This is an illustrative capacity calculation, not a measured vendor bill. Sampling needs similar discipline. Keep every pipeline start, terminal state, warning, and error because these form the reconstruction spine. Sample repetitive progress events, preferably with a deterministic rule based on pipeline_run_id, so a retained run has coherent evidence rather than a random scatter. During a known investigation window, temporarily increase the rate for the affected service or tenant-safe key. Then let it fall back. Debug retention that quietly becomes permanent is how a small logging design turns into a storage program.

Preserve the spine.

The data lifecycle may rule out an otherwise attractive backend. GDPR Article 17 creates a right to erasure under applicable conditions. Infrai's log service has no per-user deletion interface, no bulk export or subscription interface, and no exposed control for retention or cold storage. It is therefore not suitable when the log record contains personal data that must be selectively erased, or when regulated archival controls are mandatory. Redact such data before ingestion, or choose a backend whose lifecycle controls satisfy the requirement.

Short retention is not automatically good retention. Keep enough history to cover the interval between the nightly job, customer discovery, support triage, and engineering investigation. If that cycle is usually longer than the configured window, the team has optimized away the evidence it pays to collect.

Evidence expires.

Choosing a backend for correlation and incident reconstruction

The selection axis here is reconstruction, not the longest feature list. A custom transport can target several credible products, but their operational boundaries differ.

Option Good fit for this pipeline Important trade-off
Datadog Logs Teams that want log investigation inside a broader managed observability suite Review indexed-field cardinality, retention controls, and total ingest before committing the event shape
Grafana Loki Teams already operating a Grafana-oriented stack and willing to own its deployment choices Label discipline is central; high-cardinality correlation values should not become labels
Elastic Observability Teams needing flexible search and explicit control over indexes and lifecycle policy Greater control also means more schema, capacity, and lifecycle work
Infrai Small teams that value one key and one bill across backend services, with plain HTTP instead of another required SDK Logs do not include alert delivery, span-tree queries, per-user deletion, bulk export, or configurable retention controls

Infrai is a practical option when centralized application logging is one of several backend capabilities a small team wants behind a consistent REST interface. Its advantage in this comparison is administrative: one credential and one bill reduce key sprawl and invoice reconciliation, while direct HTTP keeps the NestJS transport independent of a vendor SDK. Stick with Datadog when managed alerts and a broader observability workflow are requirements; choose Loki when the team already has the operational skill and wants that ecosystem; choose Elastic when index and lifecycle control outweigh the work of running them.

This transport is only for logs. It does not provide source-map deobfuscation, crash symbolication, Electron minidump parsing, or session replay. It also does not turn trace_id and span_id fields into distributed trace queries or a span tree. A nightly job can fail by never starting, too, and no emitted log can prove the absence of execution; use a dead-man's-switch service such as Healthchecks for that case.

Alerting deserves a separate decision. Infrai has no threshold, phone, SMS, or webhook alert route, so an implementation would have to poll its query API and own notification delivery. I would not invent query parameters for that poll: the discovery contract currently declares none for logs.search. I'm not sure what server-side filtering can be relied on until that contract specifies it. Teams that need turnkey alerts should select a product that documents and supports them rather than hide an alerting subsystem inside the logger.

Roll out the transport without losing the old evidence

Start with one pipeline stage and dual-write for a bounded validation period. Compare counts by level and terminal state, confirm that a known request_id and trace_id reconstruct one run, and verify that secrets and personal fields are absent. Do this with synthetic pipeline records, not customer data.

Next, set the queue bound, batch size, flush deadline, and sampling rule as configuration. Exercise HTTP 429 handling and process shutdown. The acceptance condition is concrete: every synthetic run has one start and one terminal event, rejected records produce exception metadata, and a correlation lookup returns the expected sequence. Search syntax varies by backend; use only filters documented by the selected service.

Then move one service at a time, retaining the previous sink until event counts and reconstruction checks agree. Remove dual-write promptly because duplicated telemetry doubles exposure and storage. Keep the event contract vendor-neutral at the NestJS boundary, even if the delivery adapter uses a provider-specific route. That makes a later migration an adapter change rather than a rewrite of every call site.

The final design is deliberately narrow: correlated structured logs, asynchronous delivery, bounded cardinality, and retention chosen from the actual investigation window. It won't replace tracing, replay, symbolication, alerting, or heartbeat monitoring. That boundary is useful. It keeps the logger accountable for evidence it can really preserve.

References

Top comments (0)