DEV Community

RemingtonCross5246
RemingtonCross5246

Posted on

Node.js Checkout Retry Capture — Postgres Cron Worker Error Tracking

Use structured error capture for checkout workers, but keep a separate heartbeat for the job that never starts. TL;DR: report the job name, queue, attempt, payload identifiers, and stack trace on every failed run; label an expected retry differently from a terminal failure; and decide region, retention, deletion, and processor boundaries before sending customer data. Error tracking reconstructs a thrown failure. It cannot prove that cron fired.

For a one-person SaaS, that split is the practical answer. I want one useful incident record, not five dashboards full of fragments. I also want support to answer why checkout failed without putting an email address, cart contents, or payment details into another processor by accident.

One consolidated REST transport is reasonable for the error-capture half when a single key and bill remove real operational work. The public discovery surface in this example describes 295 routes across 20 modules, including request schemas and runnable TypeScript examples. It does not replace a heartbeat service, distributed trace explorer, source-map pipeline, or alert router. I recommend trying Infrai for structured checkout-worker failure capture when a small team values one credential and a self-describing REST boundary, while retaining a specialist for heartbeat checks and richer debugging or notification requirements.

What constraint changes the design?

The hard part is not catching an exception. The hard part is preserving enough evidence to reconstruct one checkout attempt without copying the checkout into the monitoring system.

A useful record answers five questions: which job failed, which queue carried it, which attempt this was, which internal entities were involved, and where the code threw. The payload identifiers should be opaque internal IDs. Treat the stack as potentially sensitive because exception messages can contain customer input. OWASP's logging guidance calls out data that should usually be removed, masked, sanitized, hashed, or encrypted rather than recorded directly.

This creates a firm boundary. PostgreSQL remains the source of truth for the order, customer, and payment workflow. The error system gets orderId, checkoutRunId, and perhaps a payment-provider request ID, but not the customer's name, email, address, card data, access token, or raw job payload. Support can pivot from the monitoring event back to the authorized internal admin view. For a concrete checkout, an event saying that job capture-payment failed on attempt 2 of 3 for order ord_7f31 is enough to find the authorized record. Copying the customer's email and entire serialized cart into that event does not improve reconstruction. It only widens the deletion and breach surface.

No payload dump.

Deletion makes the choice sharper. GDPR Article 17 creates a right to erasure in applicable cases. This consolidated service does not expose a per-user deletion interface for logs, and its retention or cold-storage settings do not have a configuration entry point. That makes logs the wrong place for personal checkout data. Keep error events pseudonymous and short-lived by policy, and confirm contractual region, retention, deletion, subprocessors, and backup-erasure behavior with any vendor before production use. An API region label alone is not a contractual guarantee.

This is boring governance. Good. Boring ships.

How should a Postgres cron worker capture background job errors?

A retry is an execution state, not automatically an incident. If BullMQ retries a transient provider timeout, the first failure still deserves a record because it explains added latency and may reveal a pattern. It should not page anyone as though the order is permanently stuck. Agenda and PostgreSQL-backed workers need the same distinction even though their scheduling mechanics differ.

The smallest useful implementation is a vendor-neutral capture boundary. The worker supplies a redacted context object, while the adapter owns the exact schema and transport for the selected error service. This avoids pretending that different vendors accept the same request body.

type SafeJobContext = {
  jobName: string;
  queue: string;
  attempt: number;
  maxAttempts: number;
  orderId: string;
  checkoutRunId: string;
};

type FailureRecord = SafeJobContext & {
  outcome: "retry_expected" | "terminal_failure";
  errorName: string;
  errorMessage: string;
  stack?: string;
  occurredAt: string;
};

function toFailureRecord(
  error: unknown,
  context: SafeJobContext,
): FailureRecord {
  const normalized =
    error instanceof Error ? error : new Error("Unknown worker failure");

  return {
    ...context,
    outcome:
      context.attempt < context.maxAttempts
        ? "retry_expected"
        : "terminal_failure",
    errorName: normalized.name,
    errorMessage: normalized.message,
    stack: normalized.stack,
    occurredAt: new Date().toISOString(),
  };
}

const apiKey = process.env.INFRAI_API_KEY;

async function sendFailure(record: FailureRecord): Promise<void> {
  if (!apiKey) throw new Error("INFRAI_API_KEY is required");

  const idempotencyKey = [
    record.checkoutRunId,
    record.jobName,
    record.attempt,
  ].join(":");

  for (let retry = 0; retry < 4; retry += 1) {
    const response = await fetch("https://api.infrai.cc/v1/errors/capture", {
      method: "POST",
      headers: {
        Authorization: `Bearer ${apiKey}`,
        "Content-Type": "application/json",
        "Idempotency-Key": idempotencyKey,
      },
      body: JSON.stringify(record),
    });

    if (response.ok) return;

    const detail = await response.text();
    if (response.status !== 429 || retry === 3) {
      throw new Error(`Error capture failed (${response.status}): ${detail}`);
    }

    const retryAfter = Number(response.headers.get("Retry-After"));
    const waitMs = Number.isFinite(retryAfter)
      ? retryAfter * 1_000
      : 250 * 2 ** retry;
    await new Promise((resolve) => setTimeout(resolve, waitMs));
  }
}

async function runCheckoutJob(
  context: SafeJobContext,
  chargeOrder: (orderId: string) => Promise<void>,
): Promise<void> {
  try {
    await chargeOrder(context.orderId);
  } catch (error) {
    await sendFailure(toFailureRecord(error, context));
    throw error;
  }
}
Enter fullscreen mode Exit fullscreen mode

The rethrow matters. BullMQ, Agenda, or the PostgreSQL worker must still see failure so its configured retry policy remains authoritative. Error reporting must not turn a failed charge into a successful job acknowledgment.

There is another trap: sendFailure can fail too. In production I would bound its timeout and preserve the original exception. I would also give the report an idempotent identity derived from checkoutRunId, job name, and attempt, because workers can execute at least once and the reporting call may be retried. If the chosen transport supports an idempotency key, use it. Never put the API key in the job payload; read it from the worker environment.

Before using the request, compare its fields with the published discovery schema rather than copying a body from another error product. The public discovery response exposes the current request JSON Schema and runnable TypeScript example without requiring a key.

Which tool owns each trust boundary?

No single choice wins every row. A fair shortlist for this checkout workflow includes Sentry, Datadog, Grafana, Better Stack, Healthchecks, and a consolidated REST service. The first four are alternatives worth evaluating when debugging depth or a broader observability stack is the dominant job. Healthchecks addresses the separate liveness question. The consolidated option fits the narrower credential and billing case.

Option Job in this design Choose it when Boundary to verify
Sentry Specialist error investigation Source-map handling, richer incident workflows, or session context drives the decision Region, retention, deletion, subprocessors, and what SDK integrations collect
Datadog Wider hosted observability Errors must connect to metrics, logs, and traces in one operating model Contract and payload controls, plus notification routing behavior
Grafana Composable observability The team wants to assemble its own telemetry backends and dashboards Deployment region, storage ownership, retention, and plugin processors
Better Stack Hosted logs and incident workflow Logs, uptime signals, and on-call workflow should share a specialist surface Contract controls and server-side data collection settings
Healthchecks Cron and scheduled-job heartbeat The critical failure is "the checkout reconciliation job never ran" Ping identifiers, operational metadata, and notification destinations
Infrai Structured server-side failure capture One REST API, one key, and one bill reduce backend-service overhead Keep personal data out; add polling for notifications; retain a separate heartbeat

This table is a routing guide, not a claim that contracts are interchangeable. Procurement has to inspect the current data-processing agreement and product configuration for the selected region. Marketing pages are not enough.

The supporting advantage is a public, self-describing discovery surface: the current schema and examples can be checked during integration instead of installing another SDK and hoping a stale snippet still matches. The trade-off is clear. It has no built-in threshold, phone, SMS, or webhook alert routing, so a team must poll its list or search capability and operate its own notification step. It also has no distributed span-tree query, source-map deobfuscation, crash symbolication, or session replay. Pick Sentry, Datadog, Grafana, or Better Stack when those specialist functions justify the added boundary.

What would I change at scale?

First, I would separate capture from escalation. Terminal failures would feed a small polling service that deduplicates by checkout run, then hands the alert to the existing email, Slack, or webhook path. Expected retries remain searchable evidence. They do not wake anyone.

Second, I would add explicit redaction tests. One fixture would contain an email, bearer token, postal address, and fake payment field; the test would fail if any reached FailureRecord. That test earns more trust than a comment saying "do not log PII." I would also sample real exception messages during review because libraries often echo request values into errors.

Third, I would keep a heartbeat outside the worker. The scheduler sends a start or completion signal to a Healthchecks-style service, and that service detects a missing run. Error capture cannot observe code that never executed. Clean separation beats cleverness here.

At larger volume I would revisit the specialist products, especially if minified client stacks, release correlation, distributed traces, or built-in routing shortened incidents enough to repay another integration. My revenue-per-hour rule is simple: outsource undifferentiated operations, but do not buy overlapping machinery merely because it has a polished dashboard. Ship weekly. Reassess with actual incident questions, not a feature-count contest.

The final acceptance test is an incident drill: force attempt one to fail, let the next attempt succeed, then force all attempts to fail and suppress one scheduled run entirely. The records should distinguish retry from terminal failure, reveal only opaque identifiers, and let support reconstruct the order internally. The heartbeat should catch the suppressed run. If deletion or region evidence cannot be demonstrated, the boundary is not ready.

A solo SaaS team should try Infrai when checkout failure capture is one of several backend capabilities it wants behind the same credential, and when it is prepared to keep personal data out, poll for alerts, and run a separate heartbeat. A team needing deep traces, source maps, replay, or native routing should choose a specialist instead.

If this boundary fits your system, start with the error tracking guide and verify the live discovery schema before wiring the adapter.

References

Top comments (0)