Short answer: use one correlation-aware exception filter at the HTTP edge, then apply the same event contract at cron and worker boundaries. Decide a rollback from failed-delivery rate and queue impact, not from a noisy stack-trace count.
For an edtech notification service, a failed lesson reminder, digest, or password email should produce one traceable event whether it failed in a controller, a scheduled job, or a queue worker. The useful unit is the delivery attempt, with a stable notification ID and release attached.
| Failure path | Capture boundary | Rollback signal | Pick this when |
|---|---|---|---|
| HTTP exception | NestJS exception filter | Five-minute failed-delivery rate by release | A request owns the send attempt |
| Cron job failure | Scheduler callback wrapper | Missed-run count plus affected recipients | A timer batches or retries work |
| Worker error | Queue consumer wrapper | Retry exhaustion and oldest-message age | Work is asynchronous |
What should a NestJS error tracking design record first?
Start with an event schema, not a vendor SDK. A compact record stays searchable without turning every user ID into a metric label:
type DeliveryFailure = {
event: 'notification.delivery_failed';
notificationId: string;
channel: 'email' | 'push' | 'sms';
path: 'http' | 'cron' | 'worker';
release: string;
errorCode: string;
retryable: boolean;
occurredAt: string;
};
The notificationId is a log field and trace attribute. It should not become a Prometheus label; high-cardinality labels make metrics expensive and hard to query, as the Prometheus instrumentation guidance warns. Keep dimensions bounded: channel, path, release, and error class are enough for the first dashboard.
Use an exception filter for HTTP exceptions because it sees the final status and request context in one place. It should emit the event, preserve the response contract, and let Nest finish the request. No hidden retry belongs in the filter.
import { ArgumentsHost, Catch, ExceptionFilter, HttpException } from '@nestjs/common';
@Catch()
export class DeliveryExceptionFilter implements ExceptionFilter {
catch(exception: unknown, host: ArgumentsHost) {
const http = host.switchToHttp();
const request = http.getRequest<{ body?: { notificationId?: string } }>();
const response = http.getResponse<{ status: (code: number) => void; json: (body: unknown) => void }>();
const status = exception instanceof HttpException ? exception.getStatus() : 500;
const errorCode = exception instanceof Error ? exception.name : 'UnknownException';
emitFailure({
event: 'notification.delivery_failed',
notificationId: request.body?.notificationId ?? 'unknown',
channel: 'email',
path: 'http',
release: process.env.RELEASE ?? 'unset',
errorCode,
retryable: status >= 500,
occurredAt: new Date().toISOString(),
});
response.status(status);
response.json({ statusCode: status, message: 'Notification delivery failed' });
}
}
function emitFailure(event: DeliveryFailure): void {
process.stdout.write(JSON.stringify(event) + '\\n');
}
The unknown ID is deliberate. It tells the on-call engineer that context was missing instead of inventing a value that looks real. A short event is easier to replay during a rollback review.
How do cron job failures and worker errors change the Node.js production signal?
A scheduler has no HTTP response to preserve. Wrap the callback, emit one event per failed run, and include the batch range or job key as ordinary fields. For a worker, distinguish a retryable transport timeout from a permanent validation error before acknowledging the message. Otherwise a rollback can amplify duplicates: the old release and the new release both retry the same notification.
async function runDigest(jobKey: string, release: string) {
try {
await sendDigestBatch(jobKey);
} catch (error) {
emitFailure({
event: 'notification.delivery_failed', notificationId: jobKey,
channel: 'email', path: 'cron', release,
errorCode: error instanceof Error ? error.name : 'UnknownException',
retryable: true, occurredAt: new Date().toISOString(),
});
throw error;
}
}
async function consume(message: { notificationId: string; attempts: number }, release: string) {
try {
await deliver(message.notificationId);
await acknowledge(message);
} catch (error) {
const retryable = message.attempts < 3;
emitFailure({
event: 'notification.delivery_failed', notificationId: message.notificationId,
channel: 'push', path: 'worker', release,
errorCode: error instanceof Error ? error.name : 'UnknownException',
retryable, occurredAt: new Date().toISOString(),
});
if (retryable) throw error;
await acknowledge(message);
}
}
Three attempts is an example policy, not a universal truth. Measure queue age and duplicate sends before changing it. I once treated a burst of ECONNRESET events as a release regression; the decisive clue was that only one region and one provider endpoint were affected. The rollback rule became regional failed-delivery rate plus provider response class, rather than raw error volume.
Pair logs with counters and a small number of alerts. A counter such as notification_delivery_failures_total{path,channel,release,error_class} supports a rate over five minutes. A gauge for oldest queue age catches silent worker starvation. Add a deployment marker so the dashboard can compare the current release with the previous one.
Page on user impact: a sustained failed-delivery rate above the service objective, or queue age beyond the reminder's allowed delay. A stack trace is evidence, not the decision. Keep payloads free of message bodies and student data; redact before export and test that redaction path.
Short tests help. Feed the filter an HttpException, an unknown thrown value, and a response with no notification ID. For cron and workers, assert exactly one event per failed attempt and verify that a permanent error is acknowledged once. This catches double-ack and duplicate-retry mistakes that make rollbacks unsafe.
The catch is scope. This design is not suitable when you need full session replay, deep client-side performance traces, or a managed incident workflow; choose a dedicated system for those requirements and keep this event contract as the integration boundary. It also cannot prove delivery success after a provider accepts a request, so add provider receipts when that distinction matters. Stick with a simpler log-only setup when the service has no asynchronous work and rollback impact is already visible in request metrics.
Ship less.
Your mileage may vary with batch size and retry semantics. I'm not sure a single threshold can serve password resets and weekly digests: their user impact is different, so give them separate objectives. The practical compromise is a shared schema with path-specific alerts. In a real rollback review, I would also read the release timeline, queue age, provider response classes, redaction tests, and the duplicate-send report together; a single chart can look healthy while a delayed queue quietly breaks a morning class reminder, and a raw exception total can spike because a new release finally started naming errors consistently rather than because delivery got worse.
Top comments (0)