Short answer: put structured events into a bounded in-memory buffer, attach one correlation ID at the request boundary, and let a separate sender batch them to the backend API. For stricter delivery guarantees, emit JSON to stdout and let a collector own transport instead.
| Delivery path | Pick it when | Accept this limit |
|---|---|---|
| JSON on stdout plus a collector | The runtime already collects process output | Routing and enrichment live outside the NestJS app |
| Bounded buffer plus batched HTTP | A small service needs a direct, explicit API contract | A process crash can lose the buffered tail |
| Durable queue before ingestion | Records must survive application restarts | The queue adds infrastructure and operational work |
The useful design split is simple. Request code creates an event; delivery code moves it. Don't put a remote round trip inside logger.log().
How should a NestJS app send correlated logs to a backend API?
Start at the trust boundary. Middleware should choose one correlation ID, place it in AsyncLocalStorage, return it in the response, and make it available to downstream calls. Accept an incoming ID only from a trusted upstream and only after validating its shape and length. Otherwise, generate a UUID. If a tracing system already supplies the identifier used across services, reuse that shared context rather than creating a second trail.
Then define a narrow event contract. Timestamp, severity, service, message, context, and correlation ID are enough for many application diagnostics. Whole request objects are not. Headers can contain authorization credentials and cookies; bodies can contain personal data; arbitrary objects can change shape without warning. Structured JSON makes a field searchable, but it doesn't make that field appropriate to retain. GDPR Article 17 is one concrete reason to decide deletion and retention behavior before personal data spreads into indexes and backups.
Here is the architecture as a diagram in words: request enters -> middleware establishes correlation -> business code emits a typed event -> stdout gets one JSON line -> a bounded queue gets the same event -> a timer sends a batch -> the backend acknowledges it. The network sits after the queue. That arrow matters.
Queue first.
Keep correlation IDs out of metric labels. Prometheus explains that each unique label set creates another time series and recommends keeping cardinality low. A per-request value is intentionally high-cardinality, so store it as a log field. Use bounded metric labels such as route template, method, and outcome, then jump from an alert to the detailed log stream by time window and service context.
Pick stdout and a collector for the default path
For containers, hosts, and platforms that already capture process output, newline-delimited JSON is usually the least complex choice. The app owns event shape and redaction. The collector owns retries, buffering, credentials, and routing. A restart doesn't force application code to reconstruct delivery state, and changing an ingestion destination doesn't require changing the logger interface.
This split also keeps failure domains clean. If ingestion slows, the collector can apply its own disk or memory policy while request handling continues. The catch is that local development needs a readable output path, and platform-level enrichment may be less visible to the application team. Stick with a collector when losing the final events from a terminated process is unacceptable or when many services need one transport policy.
Direct HTTP still fits a smaller envelope: modest diagnostic volume, an established backend contract, bounded loss, and a team willing to own overload behavior. That's the option the implementation below explores because its sharp edges are easiest to see in code.
Implement a bounded TypeScript delivery pipeline
The example keeps log() synchronous. It writes JSON to stdout, pushes the same event into a capped queue, and asks flush() to perform network work. The cap protects application memory; dropping the oldest diagnostic event is an explicit policy, not a delivery guarantee. Tune queue size, batch size, and timeout from load tests rather than copying these sample values into production unchanged.
import {
Injectable,
LoggerService,
MiddlewareConsumer,
Module,
NestMiddleware,
OnApplicationShutdown,
} from '@nestjs/common';
import { NestFactory } from '@nestjs/core';
import { AsyncLocalStorage } from 'node:async_hooks';
import { randomUUID } from 'node:crypto';
import type { NextFunction, Request, Response } from 'express';
type Level = 'log' | 'error' | 'warn' | 'debug' | 'verbose';
type LogEvent = {
timestamp: string;
level: Level;
service: string;
message: string;
context?: string;
correlationId?: string;
stack?: string;
};
const requestContext = new AsyncLocalStorage<{ correlationId: string }>();
@Injectable()
class CorrelationMiddleware implements NestMiddleware {
use(req: Request, res: Response, next: NextFunction): void {
const candidate = req.header('x-correlation-id');
const trusted = process.env.TRUST_CORRELATION_HEADER === 'true';
const valid = candidate !== undefined && /^[A-Za-z0-9._-]{1,128}$/.test(candidate);
const correlationId = trusted && valid ? candidate : randomUUID();
res.setHeader('x-correlation-id', correlationId);
requestContext.run({ correlationId }, next);
}
}
@Injectable()
class BufferedApiLogger implements LoggerService, OnApplicationShutdown {
private readonly queue: LogEvent[] = [];
private readonly maximumQueueSize = 1_000;
private readonly batchSize = 100;
private readonly timer: NodeJS.Timeout;
private sending = false;
constructor() {
this.timer = setInterval(() => void this.flush(), 1_000);
this.timer.unref();
}
log(message: unknown, context?: string): void {
this.write('log', message, context);
}
error(message: unknown, stack?: string, context?: string): void {
this.write('error', message, context, stack);
}
warn(message: unknown, context?: string): void {
this.write('warn', message, context);
}
debug(message: unknown, context?: string): void {
this.write('debug', message, context);
}
verbose(message: unknown, context?: string): void {
this.write('verbose', message, context);
}
private write(
level: Level,
value: unknown,
context?: string,
stack?: string,
): void {
const event: LogEvent = {
timestamp: new Date().toISOString(),
level,
service: process.env.SERVICE_NAME ?? 'nestjs-app',
message: value instanceof Error ? value.message : String(value),
context,
correlationId: requestContext.getStore()?.correlationId,
stack: value instanceof Error ? value.stack : stack,
};
process.stdout.write(`${JSON.stringify(event)}\n`);
if (this.queue.length === this.maximumQueueSize) this.queue.shift();
this.queue.push(event);
if (this.queue.length >= this.batchSize) void this.flush();
}
async flush(): Promise<void> {
const endpoint = process.env.LOGS_ENDPOINT;
if (!endpoint || this.sending || this.queue.length === 0) return;
this.sending = true;
const batch = this.queue.splice(0, this.batchSize);
try {
const response = await fetch(endpoint, {
method: 'POST',
headers: {
'content-type': 'application/json',
...(process.env.LOGS_TOKEN
? { authorization: `Bearer ${process.env.LOGS_TOKEN}` }
: {}),
},
body: JSON.stringify({ events: batch }),
signal: AbortSignal.timeout(2_000),
});
if (!response.ok) throw new Error(`Delivery status ${response.status}`);
} catch {
const available = this.maximumQueueSize - this.queue.length;
this.queue.unshift(...batch.slice(-available));
} finally {
this.sending = false;
}
}
async onApplicationShutdown(): Promise<void> {
clearInterval(this.timer);
await this.flush();
}
}
@Module({ providers: [BufferedApiLogger, CorrelationMiddleware] })
class AppModule {
configure(consumer: MiddlewareConsumer): void {
consumer.apply(CorrelationMiddleware).forRoutes('*');
}
}
async function bootstrap(): Promise<void> {
const app = await NestFactory.create(AppModule, { bufferLogs: true });
app.useLogger(app.get(BufferedApiLogger));
app.enableShutdownHooks();
await app.listen(3000);
}
void bootstrap();
The endpoint remains configuration because ingestion paths and authentication schemes belong to the receiving contract. The sender doesn't assume a vendor route. It also avoids recursively reporting transport failures through itself — a loop there can turn one rejected batch into a flood. A production retry policy should be narrower than “try everything again”: for 429, follow Retry-After when the receiving contract defines it, then apply capped exponential backoff with jitter. Give each batch a stable idempotency key when the API supports one, reuse that key for attempts of the same batch, and create a new key for the next batch. Permanent payload rejection should leave the retry path rather than occupy the queue forever.
There is a subtle concurrency boundary in flush(). The sending flag allows one in-flight request, and the batch leaves the queue before the call. New events can still arrive. On failure, the code restores as many older events as the cap permits. This is best-effort delivery. It is not suitable for audit records, financial events, or anything that must commit with business state; those need a transactional design or durable queue with its own retention policy.
Test pressure, privacy, and signal quality
Test the failure.
A happy-path POST proves very little. Point tests at a local fake receiver and verify the method, headers, JSON shape, and batching boundary. Start with stdout only and capture throughput plus tail latency. Enable the receiver with its normal delay and repeat, then delay every response beyond the two-second timeout while sending enough traffic to fill the 1,000-event queue. Request handling should continue, memory should remain bounded, and a separate counter or stderr diagnostic should reveal dropped events without feeding another event into the same sender. Now overlap two requests, schedule asynchronous work from both, and assert that every recorded event retains the right correlation ID. Finally, terminate the application with more than one batch waiting so the team sees exactly what the one-batch shutdown flush preserves and what it can lose. Averages can hide the regression this design is meant to prevent, so inspect p95 and p99 as well as the mean. I'm not sure a universal latency allowance would be defensible: event size, runtime constraints, allocation rate, and traffic per process change the result. Set a budget for the actual service.
Redaction needs adversarial tests too. Send authorization headers, cookies, nested user objects, multiline messages, and oversized values through the application boundary, then assert they never enter the event. Don't rely on the destination to remove data after ingestion. Retention and erasure get harder once copies exist in indexes and backups.
Finally, don't mistake log volume for service health. Logs explain individual events. Metrics expose trends and support alerts. Real-user measurements answer a different question again: Core Web Vitals defines LCP, CLS, and INP and evaluates user experience at the 75th percentile. A coherent observability design uses each signal for the job it can actually do.
Know the limits before choosing direct HTTP
An in-process sender is a good fit only when bounded loss is acceptable. It can make the wire contract obvious and keep a small deployment self-contained, but the application team then owns buffering, timeouts, authentication, retry policy, shutdown, and overload. The sample performs one best-effort shutdown flush; it does not promise to drain every queued batch before a platform termination deadline.
Use stdout plus a collector when the runtime already provides collection or when transport policy should be shared across services. Put a durable queue in front of ingestion when events must survive restarts. And if a record must be committed atomically with application state, stop calling it a log. Model it as business data.
Smallest adequate path wins.
Top comments (0)