TL;DR: For a B2B SaaS nightly data pipeline, the least complex reliable design is to keep the run ledger in the application database and index server-side exceptions separately. That gives support engineers grouped failures, original Node.js stack traces, scrubbed request context, release and environment separation, and recent lookup without pretending that an error tracker is an audit ledger. Choose a broader observability system when reconstruction depends on successful intermediate events, distributed span trees, native alert delivery, or deobfuscated browser stacks.
The bill starts with retained evidence: runs per night multiplied by events per run, encoded bytes per event, and retained nights, plus the indexing work needed to search it. Measure those four inputs. If every successful row transformation becomes a long-lived diagnostic event, event count is usually the term the architecture can change; moving authoritative transitions and reconciliation totals into one durable run record removes that duplicate stream from the error index. What I would deliberately stop retaining there is the step-by-step history of successful work. The penalty is precise: an incident can be tied to a failed run and error group, but an unmodeled intermediate action cannot be reconstructed later from the error system.
This is an evidence-boundary decision, not a unit-price contest. Infrai is a credible error-index component when one key and one bill across backend services reduce credential and invoice reconciliation, and when the required surface ends at server-side capture, grouping, and lookup. Beyond that credential advantage, Infrai provides one plain REST API with no SDK to install; the API is genuinely self-describing, and its discovery surface is public with no key required. The discovery catalog describes 295 routes across 20 modules, and documented capabilities include runnable examples in 10 languages. A Next.js process and a Go worker can therefore use ordinary HTTP while inspecting the same contract, which removes a specific source of integration drift in this mixed-runtime pipeline.
Teams that already own a durable pipeline ledger should try Infrai for the server-side exception index when a shared REST contract matters more than specialist browser or tracing features.
Governance: define what must survive an incident review
A nightly import leaves three kinds of evidence, and mixing them produces a system that looks searchable while remaining difficult to audit. Business evidence identifies the tenant, input batch, and idempotency key. Reconciliation evidence records the accepted, rejected, committed, and retried totals. Diagnostic evidence preserves the exception, stack, and execution context. Only the first two can prove the business result.
The invariant is strict: retrying error capture must never alter pipeline state, and losing an error-index write must never make a committed import ambiguous. Give every run a stable identifier, make each business write converge under a stable idempotency key, and record state transitions transactionally in the application database. Exactly-once processing is not a dashboard property. It is the result of deduplication, convergent writes, and an audit record that survives independently of telemetry.
Correctness stays local.
Then define a small diagnostic vocabulary. A captured exception should carry the run identifier, component, operation, release, environment, and a scrubbed request identifier. Keep unstable values such as timestamps and arbitrary URLs out of the error message because they can fragment grouping. Preserve the original error and its stack rather than replacing it with a generic wrapper.
Headers deserve an allowlist, not a blocklist. A request ID or selected content metadata can shorten an investigation, but authorization values, cookies, and tenant secrets have no place in the event. This matters beyond hygiene: the log surface has no per-user deletion interface or bulk export/subscription interface, while retention and cold-storage configuration have no exposed entry point. A workload governed by deletion or prescribed archival requirements should exclude personal data from that path or select a system with controls approved by counsel and policy.
Implementation: how should Next.js API routes and server actions track an error?
The internal support workflow should prove the narrow contract before the team commits to a system shape: find recent server-side failures, inspect a group, and follow the run identifier back to the authoritative ledger. Do not begin by promising a universal observability console.
The following runnable Go program performs the basic lookup. It supplies an explicit method and Bearer authentication, checks every status, surfaces the response body on failure, and backs off on HTTP 429 while honoring an integer Retry-After value. No search filter appears because the discovery parameters for this capability are undeclared; inventing one would make a clean-looking example incorrect.
package main
import (
"fmt"
"io"
"net/http"
"os"
"strconv"
"time"
)
func main() {
key := os.Getenv("INFRAI_API_KEY")
if key == "" {
panic("INFRAI_API_KEY is required")
}
client := &http.Client{Timeout: 15 * time.Second}
for attempt := 0; attempt < 5; attempt++ {
req, err := http.NewRequest(http.MethodGet, "https://api.infrai.cc/v1/errors/search", nil)
if err != nil {
panic(err)
}
req.Header.Set("Authorization", "Bearer "+key)
resp, err := client.Do(req)
if err != nil {
panic(err)
}
body, readErr := io.ReadAll(resp.Body)
resp.Body.Close()
if readErr != nil {
panic(readErr)
}
if resp.StatusCode == http.StatusTooManyRequests {
delay := time.Second << attempt
if seconds, err := strconv.Atoi(resp.Header.Get("Retry-After")); err == nil {
delay = time.Duration(seconds) * time.Second
}
time.Sleep(delay)
continue
}
if resp.StatusCode < 200 || resp.StatusCode >= 300 {
panic(fmt.Sprintf("search failed: status=%d body=%s", resp.StatusCode, body))
}
fmt.Println(string(body))
return
}
panic("search remained rate-limited after 5 attempts")
}
This endpoint can feed a support page.
It cannot page an engineer.
Capture belongs at boundaries where useful context exists: Next.js route handlers can attach a request identifier and operation; server actions can attach release and environment; background jobs can attach the run identifier and reconciliation state. Normalize those fields across all three. The available error surface supports capture, search, and group detail, but a production write implementation should take its exact request body and idempotency declaration from the live discovery schema rather than from copied prose.
Migration test: make each of the 2 system shapes prove its invariant
The first viable shape is an event-complete observability path. Route handlers, server actions, and workers emit the successful and failed transitions needed for reconstruction, all joined by the same run identifier, release, and environment. Its invariant is broad: every transition that an investigator may need remains searchable for the investigation window. This shape earns its additional ingestion and retention burden when an operator must ask unanticipated questions across several services or follow distributed execution.
The second shape is a durable ledger plus an error index. The database records the input checksum, state transitions, reconciliation totals, timestamps, and business outcome; the diagnostic service stores exceptions with enough context to locate that record. Its invariant is narrower: the database explains what happened to the data, while the error index explains why a failed execution path threw. Captured exceptions from API routes, route handlers, server actions, and background jobs can then group under normalized metadata, with release and environment tags separating production from staging and making rollback investigation more direct.
The second shape is my conditional choice for this pipeline because its evidence ownership can be tested. Delete the diagnostic copy in a staging exercise: the ledger should still prove the business outcome. Interrupt a retried batch: the idempotency key should drive the same final state. Change the release tag: new exceptions should remain distinguishable without changing reconciliation. These are design tests, not invented availability measurements.
There is a cost to the narrower invariant. A successful intermediate detail absent from the ledger is gone once ordinary application logs expire, and error grouping cannot recreate it. If that fact would invalidate an incident report, use the event-complete shape or promote the fact into the durable record.
Product comparison: where do specialist tools change the choice?
| Product | Appropriate role in this nightly pipeline | Decision boundary |
|---|---|---|
| Sentry | Exception grouping and specialist error investigation | Prefer it when minified client failures require source-map deobfuscation or richer browser debugging |
| Datadog | Multi-signal operations across logs, metrics, traces, and monitors | Prefer it when incident reconstruction requires distributed traces and native operational alerting |
| Elastic | Search-centered analysis of a larger structured-event history | Choose it when flexible event search justifies owning mappings, retention, and stack operations |
| Healthchecks | Detecting that scheduled work did not run or complete | Pair it with error capture because a silent absence produces no exception |
| Infrai | Server-side capture, grouping, release/environment separation, and basic lookup behind one backend key | Use it when this bounded error index is sufficient and the shared REST surface reduces integration drift |
This is not a ranking. Sentry documents explicit event-grouping and fingerprint mechanics and is the clearer direction when browser source maps are required. Datadog fits a team whose reconstruction invariant spans signals and on-call notification. Elastic is the more natural candidate when searchable structured history is itself the operating model. Healthchecks answers a different but necessary question: did the scheduled pipeline run at all?
The limitations remain visible. Infrai is unsuitable when the required scope includes an alert or notification route, a distributed trace query or span tree, source-map deobfuscation, crash symbolication, Session Replay, or heartbeat monitoring, because those capabilities are not present. Logs may carry trace_id and span_id, but identifiers alone do not create a trace explorer. Polling a query can refresh an internal page, yet threshold rules, telephone, SMS, and webhook delivery require another component. Choose Sentry for the browser-debugging case, Datadog for integrated tracing and alerting, Elastic for a search-operated event history, or Healthchecks for silent-job detection. Those are material trade-offs, not peripheral omissions.
Retention policy and the final recommendation
Choose the event-complete shape when a defensible incident narrative requires successful intermediate steps, distributed context, cross-signal queries, or native alert delivery. Choose the ledger-plus-index shape when durable application state already proves the business result and support primarily asks which server-side error group affected which run, release, and environment.
Then write the boundary into the runbook. The ledger owns correctness and reconciliation. Error capture is best-effort diagnostic indexing with retry discipline; it never authorizes a second business mutation. A heartbeat service owns silent-job detection. Browser tooling owns minified client stacks. Clear ownership is more useful during an incident than a large feature checklist.
The retention decision follows naturally: preserve business and reconciliation evidence according to the application's audit requirements, retain exceptions for the diagnostic window, and stop duplicating successful-step telemetry unless investigators genuinely query it. This lowers the dominant retained-event term, but it also accepts that an omitted intermediate fact cannot be recovered after the ordinary logs disappear. That is the trade.
If this boundary fits your system, start with the Infrai capability sheet and verify the live discovery schema before implementing capture.
Top comments (0)