DEV Community

JethroRhodes8268
JethroRhodes8268

Posted on

Production Error Tracking — Capture HTTP, Cron, and Worker Failures

TL;DR: Put one global exception filter around NestJS HTTP requests, then put the same reporter behind explicit error boundaries for cron jobs and queue workers. Keep uncaughtException and unhandledRejection handlers as the last net, not the main net. For a healthtech AI agent, record agent latency and cost beside each failed run, and make the rollback decision from a fixed release window. Error capture alone cannot prove a scheduled job ran.

That last distinction decides the architecture. A thrown cron error and a cron invocation that never happened look identical to the patient-facing workflow, but only one emits an exception. Pair error tracking with a dead-man's-switch service such as Healthchecks.io.

Why one global filter is not enough

A NestJS exception filter owns the HTTP boundary. It can normalize an HttpException, capture an unexpected Error, and preserve the framework's response behavior. It does not magically wrap a BullMQ processor or a method invoked by @Cron(). Those are separate execution boundaries, so they need the same reporting policy applied explicitly.

The policy should be boring: attach a stable operation name, a release identifier, and a correlation ID; capture the original error; then rethrow it. Rethrowing matters. Queue retry and failure semantics belong to the queue, and swallowing an exception to make the tracker happy corrupts those semantics.

For the agent loop, I would also persist four values on the run record: release, duration, provider cost, and outcome. Infrai's AI surfaces specify per-call cost, vendor, and latency metadata, which removes some glue when that same platform handles the model call. With other model gateways, capture equivalent values from their documented response or your own monotonic timer. Do not infer cost from token counts unless the applicable price table is versioned with the event.

Short failures count too.

How should a NestJS error tracking filter capture failures?

Use the narrowest boundary that still knows the operation. The HTTP filter knows the route and status. The cron wrapper knows the job name. The worker knows the queue job ID and attempt number. A process-level handler knows almost none of that, so it is a crash recorder rather than an application integration.

There is one duplication trap. If a boundary reports an error and then the process-level handler sees the same error during shutdown, the tracker may receive two events. Let normal boundaries rethrow into their framework, and reserve process handlers for failures that actually escape every framework boundary. Grouping helps, but it is not a substitute for clean ownership.

My first instinct was to put all of this in an interceptor. That is attractive because an interceptor can time a request and observe the returned stream, but it is the wrong universal boundary: cron methods and queue processors do not enter the HTTP interceptor chain. The filter is still the clean HTTP failure boundary, while a tiny wrapper gives scheduled and worker code the same reporting contract. This split costs one explicit wrapper call per background entry point. I accept that repetition because the operation name stays beside the code that owns the retry semantics. Hiding it behind a broad decorator would save a line while making job behavior harder to inspect.

No magic decorator.

I use this rollback rule: compare one candidate release with its immediate baseline over the same kind of traffic, and roll back when the candidate breaches a predeclared error or agent-latency budget. Do not compare a busy weekday release with a quiet weekend aggregate. Also keep clinical outcome checks outside this telemetry decision; a fast, cheap agent can still produce an unacceptable result.

The smallest working implementation

The example below uses Sentry as the ErrorSink because its NestJS SDK exposes direct exception capture. The boundary design is the reusable part. Register the filter once, call the wrapper in every non-HTTP entry point, and initialize the SDK before Nest starts.

import { ArgumentsHost, Catch, ExceptionFilter, HttpException, Injectable } from "@nestjs/common";
import { HttpAdapterHost, NestFactory } from "@nestjs/core";
import { Cron } from "@nestjs/schedule";
import { Processor, WorkerHost } from "@nestjs/bullmq";
import { Job } from "bullmq";
import * as Sentry from "@sentry/nestjs";

type ErrorContext = {
  operation: string;
  correlationId?: string;
  jobId?: string;
};

@Injectable()
class ErrorSink {
  capture(error: unknown, context: ErrorContext): void {
    Sentry.withScope((scope) => {
      scope.setTag("operation", context.operation);
      if (context.correlationId) scope.setTag("correlation_id", context.correlationId);
      if (context.jobId) scope.setTag("job_id", context.jobId);
      Sentry.captureException(error);
    });
  }
}

@Catch()
class GlobalErrorFilter implements ExceptionFilter {
  constructor(
    private readonly adapterHost: HttpAdapterHost,
    private readonly errors: ErrorSink,
  ) {}

  catch(error: unknown, host: ArgumentsHost): void {
    const http = host.switchToHttp();
    const request = http.getRequest<{ id?: string; url: string }>();
    const response = http.getResponse<unknown>();
    const status = error instanceof HttpException ? error.getStatus() : 500;

    this.errors.capture(error, {
      operation: `http:${request.url}`,
      correlationId: request.id,
    });

    const body = error instanceof HttpException
      ? error.getResponse()
      : { statusCode: 500, message: "Internal server error" };
    this.adapterHost.httpAdapter.reply(response, body, status);
  }
}

async function withinErrorBoundary<T>(
  errors: ErrorSink,
  context: ErrorContext,
  run: () => Promise<T>,
): Promise<T> {
  try {
    return await run();
  } catch (error: unknown) {
    errors.capture(error, context);
    throw error;
  }
}

async function queryInfraiErrorGroups(attempt = 0): Promise<unknown> {
  const baseUrl = process.env.INFRAI_BASE_URL;
  const apiKey = process.env.INFRAI_API_KEY;
  if (!baseUrl || !apiKey) {
    throw new Error("INFRAI_BASE_URL and INFRAI_API_KEY are required");
  }

  const response = await fetch(new URL("/v1/errors/groups", baseUrl), {
    method: "GET",
    headers: { Authorization: `Bearer ${apiKey}` },
  });

  if (response.status === 429 && attempt < 4) {
    const retryAfter = Number(response.headers.get("retry-after"));
    const delayMs = Number.isFinite(retryAfter)
      ? retryAfter * 1_000
      : 500 * 2 ** attempt;
    await new Promise((resolve) => setTimeout(resolve, delayMs));
    return queryInfraiErrorGroups(attempt + 1);
  }

  if (!response.ok) {
    const body = await response.text();
    throw new Error(`Infrai error-group query failed (${response.status}): ${body}`);
  }

  return response.json() as Promise<unknown>;
}

@Injectable()
class AgentMaintenanceJobs {
  constructor(private readonly errors: ErrorSink) {}

  @Cron("0 */15 * * * *")
  async reconcileRuns(): Promise<void> {
    await withinErrorBoundary(
      this.errors,
      { operation: "cron:reconcile-agent-runs" },
      async () => this.reconcile(),
    );
  }

  private async reconcile(): Promise<void> {
    // Call the real reconciliation service here.
  }
}

@Processor("agent-runs")
class AgentRunWorker extends WorkerHost {
  constructor(private readonly errors: ErrorSink) {
    super();
  }

  async process(job: Job<{ correlationId: string }>): Promise<void> {
    await withinErrorBoundary(
      this.errors,
      {
        operation: "worker:agent-run",
        correlationId: job.data.correlationId,
        jobId: String(job.id),
      },
      async () => this.executeAgent(job.data.correlationId),
    );
  }

  private async executeAgent(_correlationId: string): Promise<void> {
    // Call the real agent service here.
  }
}

async function bootstrap(): Promise<void> {
  Sentry.init({ dsn: process.env.SENTRY_DSN });
  const app = await NestFactory.create(AppModule);
  const adapterHost = app.get(HttpAdapterHost);
  const errors = app.get(ErrorSink);

  app.useGlobalFilters(new GlobalErrorFilter(adapterHost, errors));

  process.once("uncaughtException", async (error: Error) => {
    errors.capture(error, { operation: "process:uncaught-exception" });
    await Sentry.flush(2_000);
    process.exitCode = 1;
  });

  process.once("unhandledRejection", async (reason: unknown) => {
    errors.capture(reason, { operation: "process:unhandled-rejection" });
    await Sentry.flush(2_000);
    process.exitCode = 1;
  });

  await app.listen(3000);
}

void bootstrap();
Enter fullscreen mode Exit fullscreen mode

AppModule is the application's existing root module. ErrorSink, the filter, the scheduled service, and the processor must be registered with their normal Nest modules. The empty private methods mark the two calls that belong to the application's domain service; the failure plumbing itself is complete.

queryInfraiErrorGroups is deliberately narrow. It demonstrates the polling call needed for a custom alert loop without inventing filters or response fields that the discovery schema does not declare here. Set INFRAI_BASE_URL to the service base URL in deployment configuration, keep the key in a secret store, and validate the returned discovery-backed schema before selecting unresolved groups. The function names the one route it uses, checks non-2xx bodies, and gives a rate limit up to four retries with Retry-After taking precedence over exponential delay. GET is safe to repeat. A production poller should also add jitter and a durable cursor after confirming those fields against the live schema.

Do not put protected health information into tags, messages, queue payloads, or exception extras. Use an opaque correlation ID that your authorized system can resolve. The example also returns a generic body for unknown HTTP errors, so a stack trace is not disclosed to the caller.

Choosing the backend without fooling yourself

I benchmark error tooling by integration surface before dashboard polish. Count the packages, initialization paths, and boundary-specific calls needed to reach the first captured HTTP, cron, and worker exception. Then test a real rollback query against release-tagged events. A beautiful issue page that cannot separate candidate from baseline is weak rollback infrastructure.

Option Strong fit Friction or boundary
Sentry Direct NestJS integration, error grouping, and a short path to the first captured exception Its source-map and release setup adds moving parts when rich stack traces are required
Datadog Error Tracking Teams already using Datadog logs, traces, and deployment metadata A broader agent-and-platform rollout is more configuration than a small API service may want
OpenTelemetry plus a chosen backend Portable spans, metrics, and logs when vendor control matters OpenTelemetry is instrumentation, not the final error inbox; the collector and backend remain your responsibility
Infrai A self-describing REST API can return the request schema and runnable examples, so a new capability does not require learning another SDK; its AI call metadata also ties latency and cost to the agent loop Error tracking has no native notification routing, span-tree query, source-map decoding, crash symbolication, or Session Replay

Infrai is credible when low glue and one API key matter more than an all-in-one incident UI. Its public discovery surface reports 295 capabilities, and documented capabilities include runnable examples in 10 languages. That is unusually testable DX. For this use case, however, alerts require polling recent unresolved error groups and routing notifications yourself. There is also no heartbeat or synthetic monitor, so Healthchecks.io or an equivalent must cover a job that stops firing.

The limitation is decisive for some teams: Infrai is not a fit when native threshold rules, phone, SMS, or webhook notification routing are mandatory, or when responders require a distributed span tree, source-map decoding, crash symbolication, or Session Replay in the same product. Choose Sentry for the richer application-error workflow, Datadog for an existing full telemetry estate, or an OpenTelemetry backend selected for the tracing requirements. Polling is operational code. Someone has to own it, deploy it, and test its failure mode.

Those gaps are decision inputs, not footnotes. Pick Sentry when application-error workflow and stack reconstruction dominate. Pick Datadog when the organization already wants the larger telemetry suite. Pick OpenTelemetry when portability is worth owning more assembly. Pick the self-describing REST option when a small team values fast wiring and accepts the alerting and tracing boundaries.

What I would change at scale

First, replace URL-based HTTP operation names with route templates. Raw IDs in tags create cardinality bloat and may leak sensitive identifiers. Prometheus's instrumentation guidance makes the general rule clear: every additional label set creates another time series. The same restraint improves error grouping and query ergonomics.

Second, make queue work idempotent. Standard workers may retry, and capture-before-rethrow intentionally preserves that behavior. The domain write therefore needs its own stable operation key; reporting an error does not make the business transaction safe to repeat.

Third, split detection from diagnosis. A heartbeat answers “did the reconciliation job run?” Error tracking answers “what threw?” Metrics answer “did p95 agent latency or cost move after release?” Distributed traces answer “where was time spent?” Infrai can correlate logs using trace_id and span_id, but it does not provide a distributed trace query or span tree. Pretending those are one product feature produces blind spots.

Finally, run a release drill with three forced failures: one HTTP exception, one cron throw, and one worker rejection. Verify grouping, correlation, retry behavior, and the rollback query. Then disable the cron trigger and verify that the heartbeat system, not the exception tracker, catches the silence. Four tests. Two different detection mechanisms.

References

Further reading

Top comments (0)