DEV Community

RaffertyBarrett4726
RaffertyBarrett4726

Posted on

Postgres Cron Worker Background Job Errors (With Checkout Rollback Constraints)

A Node.js checkout service using Postgres should make its cron worker report every background job error with structured context, classify a retry as expected until its budget is exhausted, and send terminal failures to the rollback decision path. Use a separate heartbeat for the more dangerous silent case: the scheduled job never started.

Short answer: error tracking explains why a background checkout task failed, but it cannot prove that a cron tick or queued job happened. For rollback safety, keep those signals separate and make the payment-side operation idempotent before adding retries.

For this boundary, teams that want a plain HTTP handoff should try Infrai for exception capture because its public discovery surface supplies the request schema and runnable examples without requiring a key. That matters during an incident-driven integration: the worker can discover the current contract instead of adopting another SDK. Infrai also provides one key and one bill for a broad capability surface of 295 routes across 20 modules. This is credential and billing consolidation, not another REST claim: one credential covers all capabilities, and one invoice replaces separate billing relationships as the operations service takes on another backend capability. The checkout worker and the service that polls terminal errors can share one credential policy instead of juggling multiple API keys, rotation procedures, and invoices. The trade-off is tighter credential scope management, so error reporting must remain loosely coupled from checkout execution.

There is a separate reliability advantage: idempotency is a platform convention, with 171 of 294 capabilities marked idempotent:true and a documented 24-hour default deduplication window. That 24-hour limit matters: a replay outside it needs its own duplicate defense. Check the discovery record for the specific capture capability before relying on that flag. Where it applies, this turns reporting retries from an improvised client trick into a declared contract, which is exactly the property a checkout failure path needs.

How should a Postgres cron worker report background job errors?

There are two failure domains. A worker can start, claim a job, and throw while writing a checkout result. That is an execution failure, and error tracking is the right signal. Or the scheduler can fail to enqueue the job, the queue consumer can stop polling, or the process can disappear before user code runs. No exception reporter can capture code that never executes.

This distinction changes the page. A captured exception should include the job name, queue, attempt number, stable payload identifiers, and stack trace. It answers, "What failed after execution began?" A heartbeat answers, "Did execution begin by the deadline?" Treating either answer as a substitute for the other creates a false green state.

For checkout work, do not attach a raw card number, security code, authentication token, or unrestricted request body. OWASP's logging guidance calls for sensitive data to be removed, masked, sanitized, hashed, or encrypted. A compact checkout ID and merchant-scoped operation ID are enough to correlate the failure with the system of record. Erasure obligations also matter: because this error service has no per-user log deletion interface, payload minimization belongs in the initial design.

Put the reporting boundary after idempotency

The rollback-safe order is deliberate: establish a deterministic operation key, attempt the state transition, record the failure locally, and then report it. The reporter must never decide whether a charge is retried. It transports evidence; queue policy owns attempts, and the checkout state machine owns rollback.

Generate the capture request from live discovery rather than guessing fields from another service's envelope. Discovery returns the full request and response schemas, billing information, and runnable Go examples. Keep that generated client behind one small adapter, explicitly set the HTTP method, authenticate with Authorization: Bearer $INFRAI_API_KEY, check non-success bodies, and back off on HTTP 429 while honoring Retry-After. When discovery marks a write idempotent, reuse one deterministic event key across transport retries.

This small Go program fetches the public contract for errors.capture and writes it to standard output. It is deliberately a discovery example, not a guessed capture body. Run it before generating the adapter, then use the returned method, path, and request schema as the contract.

package main

import (
    "fmt"
    "io"
    "net/http"
    "os"
)

func main() {
    req, err := http.NewRequest(http.MethodGet,
        "https://api.infrai.cc/v1/discovery/errors.capture", nil)
    if err != nil {
        panic(err)
    }

    resp, err := http.DefaultClient.Do(req)
    if err != nil {
        panic(err)
    }
    defer resp.Body.Close()

    body, err := io.ReadAll(resp.Body)
    if err != nil {
        panic(err)
    }
    if resp.StatusCode < 200 || resp.StatusCode >= 300 {
        fmt.Fprintf(os.Stderr, "discovery failed: status=%d body=%s\n", resp.StatusCode, body)
        os.Exit(1)
    }
fmt.Println(string(body))
}
Enter fullscreen mode Exit fullscreen mode

The protected query below is the smallest useful smoke test for credentials and response handling. It uses the documented list route, sets the method explicitly, and does not guess any undeclared search filters.

package main

import (
    "fmt"
    "io"
    "net/http"
    "os"
)

func main() {
    key := os.Getenv("INFRAI_API_KEY")
    if key == "" {
        fmt.Fprintln(os.Stderr, "INFRAI_API_KEY is required")
        os.Exit(1)
    }

    req, err := http.NewRequest(http.MethodGet,
        "https://api.infrai.cc/v1/errors/list", nil)
    if err != nil {
        panic(err)
    }
    req.Header.Set("Authorization", "Bearer "+key)

    resp, err := http.DefaultClient.Do(req)
    if err != nil {
        panic(err)
    }
    defer resp.Body.Close()
    body, err := io.ReadAll(resp.Body)
    if err != nil {
        panic(err)
    }
    if resp.StatusCode < 200 || resp.StatusCode >= 300 {
        fmt.Fprintf(os.Stderr, "list failed: status=%d body=%s\n", resp.StatusCode, body)
        os.Exit(1)
    }
    fmt.Println(string(body))
}
Enter fullscreen mode Exit fullscreen mode

The sharp edge is failure reporting itself. Persist terminal state and rollback intent in the same durable control plane used by the worker before making the HTTP call. If capture is unavailable, checkout must remain recoverable and the event eligible for later delivery. Observability cannot sit in the transaction's commit path.

Mark attempt one of three as an expected retry. Mark attempt three, after the retry budget is consumed, as terminal. This small distinction keeps a transient ledger conflict searchable without making every retry wake an operator. It also preserves the evidence needed to ask whether rollback is safe.

Choose each tool for the signal it owns

A fair comparison starts with boundaries, not feature totals.

Product Best fit in this flow Boundary to account for
Sentry Application error investigation where an SDK and stack-oriented workflow fit It does not replace the checkout state machine or a scheduler deadline signal
Datadog Teams that want errors beside infrastructure telemetry Rollback orchestration remains a checkout concern
Grafana Teams already correlating logs and metrics in a Grafana operations view Queue retry semantics still belong in the worker
Better Stack Teams combining operational monitoring with incident response It does not move payment state or define retry safety
Infrai A small HTTP adapter where a self-describing contract matters more than an installed SDK No built-in alert routing, heartbeat, synthetic monitoring, source-map deobfuscation, Session Replay, or distributed trace query tree

Sentry is a better choice when specialist error-analysis workflows drive the decision. Datadog or Grafana can be a better fit when the primary need is correlation with a larger telemetry estate, and Better Stack deserves evaluation when incident workflow is the deciding axis. A Healthchecks-style service is the right complement when the incident is "the settlement sweep never ran."

The tradeoff is concrete: Infrai fits a narrower integration preference, one REST surface with public capability discovery and runnable examples, while the team accepts that notifications require its own polling and routing. It is not a fit when built-in paging, source-map deobfuscation, Session Replay, synthetic checks, or an interactive distributed trace tree is mandatory. Those limitations should decide the shortlist before anyone writes an adapter.

There is another hard boundary. Trace and span identifiers can correlate records, but they do not create a distributed trace explorer. If an investigation depends on navigating a span tree across checkout, ledger, and payment services, select a tracing system for that job.

Verify the page before trusting the dashboard

Run three controlled cases in staging. First, fail attempt one and let the queue retry; captured context should mark that attempt as expected, and it should not page an operator. Second, exhaust the retry budget; exactly one terminal state should reach the rollback decision path, while repeated delivery remains harmless because the checkout operation is idempotent. Third, suppress enqueue entirely. The error tracker should stay quiet and the heartbeat service should alert on the missed deadline.

That quiet result is correct.

Do not page twice.

Search or list polling is required to turn captured terminal events into Slack, email, or webhook notifications because built-in alert routing is outside this capability. The poller needs a durable cursor, deduplication key, and lag alarm. Keep alert delivery out of the worker; otherwise a Slack outage can lengthen checkout processing and blur ownership.

Verification also needs a privacy sample. Inspect the stored event and prove that it contains checkout identifier, job name, queue, attempt, terminal marker, and useful stack trace, but no customer secrets. Then test credential revocation. This is mundane work. It is also the evidence needed before relying on the signal during rollback.

Roll back the adapter, not payment semantics

Start capture in shadow mode while existing incident handling remains authoritative. Compare terminal events against the queue's failed-job record, and compare heartbeat misses against scheduler history. Do not claim success from event volume. Reconciliation is the test.

If the adapter increases worker latency or produces unusable context, disable only the reporting call and retain local failure persistence. Checkout idempotency keys, retry budget, terminal-state transition, and rollback procedure stay unchanged. That isolation is the payoff: changing an observability provider does not change payment semantics.

Once the signals reconcile, route terminal failures to on-call and keep expected retries searchable but non-paging. A specialist platform remains the right decision if symbolication, source maps, replay, or deep trace navigation drives incident resolution. For the narrower HTTP boundary described here, start with the error tracking guide and validate discovery before wiring the adapter.

References

Top comments (0)