2026 Scheduled Import Observability: Proving Results Before Assigning Operational Cost
The constraint changes the design: a scheduled e-commerce import that produces no result may never throw, so a Node.js error-tracking library cannot prove that the job ran. Use the library for exceptions, add an explicit completion record with a stable run identifier, and have a separate heartbeat check detect a missing result before assigning the operational cost to a source or cost center.
Short answer: initialize Sentry, Rollbar, or a similar library before application work, capture uncaught and HTTP errors, and wrap the import boundary for explicit reporting. Then record successful and empty outcomes separately. A missed schedule needs a Healthchecks-style monitor or an equivalent polling check; it is not an error event.
The distinction is easy to miss.
What should the import system prove?
For every scheduled import, I want one answerable question: did the expected result arrive before the business deadline? The completion record should carry job_name, run_id, source, started_at, finished_at, rows, and cost_center. The last field is deliberate. If marketplace, catalog, and inventory feeds share workers, an error count cannot explain whose operational cost is rising.
There are three useful outcomes:
-
failed: the worker reported an exception or an HTTP failure. -
completed_empty: the run finished, but produced zero rows where zero is suspicious. -
missing: no completion signal arrived before the deadline.
Only the first outcome is naturally visible to an error-tracking SDK. The other two require an application event and a deadline check. OpenTelemetry describes logs as a signal whose records can carry attributes, which is a useful model for correlating error events, completion events, and deadline checks without pretending they are one signal.
Keep the business cost calculation outside the error tracker. A zero-row result may be a legitimate supplier response, a broken credential, or a scheduler that never fired. The tracker preserves evidence; a small attribution job applies the business rule after the run state is known.
Why does a missing import need another signal?
Absence is not an exception. Consider a nightly catalog import expected by 02:00 UTC. The worker may never start, wait behind a stuck process, or exit before it initializes the library. In all three cases, a design built only around try/catch has no event to send.
The minimum design is a pair of writes: emit an error when a failure is observed, and emit a completion event after the durable result is committed. A scheduler or Healthchecks-style service checks for the latter. Give that check a tolerance window. A deadline that is too strict converts ordinary queue delay into an incident storm, while one that is too generous delays cost attribution.
Use the same run_id in error context, completion log, queue message, and cost ledger. Trace and span fields can help correlate records when available, but this design does not depend on a distributed-trace query or a span tree.
No exception is required.
That is the awkward part. A catalog feed can return a valid response, write zero rows, and leave the error tracker quiet; the system must retain the schedule, source, row count, deadline, and cost center long enough for an attribution job to distinguish a legitimate empty feed from a missed run. I would make that distinction in the workflow, because a library cannot infer whether zero is normal for one supplier on one day. The import worker should also rethrow after reporting an exception. Swallowing it makes the scheduler believe the run succeeded, which corrupts both operational status and cost attribution.
The application contract can stay provider-neutral. This Python example shows the capture boundary for a REST capability; the Node.js worker can implement the same contract through its chosen SDK or HTTP client. The operational limits matter too: use a 10-second request timeout, make four bounded attempts, and treat HTTP 429 as a retryable response rather than a reason to loop forever. Those are explicit client choices, not promises about provider latency.
import os
import time
import uuid
import requests
def capture_error(error_payload):
api_key = os.environ["INFRAI_API_KEY"]
base_url = os.environ["INFRAI_BASE_URL"].rstrip("/")
response_url = f"{base_url}/v1/errors/capture"
headers = {
"Authorization": f"Bearer {api_key}",
"Content-Type": "application/json",
"Idempotency-Key": str(uuid.uuid4()),
}
for attempt in range(4):
response = requests.post(
response_url,
headers=headers,
json=error_payload,
timeout=10,
)
if response.status_code != 429:
if not response.ok:
raise RuntimeError(
f"error capture failed: {response.status_code} {response.text}"
)
return response.json()
retry_after = response.headers.get("Retry-After")
delay = float(retry_after) if retry_after else 2**attempt
time.sleep(delay)
raise RuntimeError("error capture rate limit did not clear after retries")
The key comes from an environment variable, the method is explicit, a 429 honors Retry-After, and non-success responses remain visible. The idempotency key prevents a retry after a network timeout from recording the same failure twice. Do not send this authorization header to a separate URL returned for another operation.
Which tools fit the evidence?
The comparison should follow the missing proof, not the popularity of a dashboard. Sentry is a natural fit when the main requirement is SDK-based exception capture and context. Rollbar belongs in the same shortlist for application error monitoring. Datadog is a broader observability choice when error records need to sit beside other operational telemetry. None of those labels, by themselves, answers “did this scheduled import produce a result?”
| Option | Strong fit | Boundary for silent imports | Cost attribution implication |
|---|---|---|---|
| Sentry | Capturing application exceptions with project context | A missing completion still needs a positive event and a deadline check | Add cost_center and run_id to captured context, then calculate attribution outside the error stream |
| Rollbar | Error monitoring for application failures | A worker that never emits an error remains invisible | Keep the run ledger separate so a low error count cannot be mistaken for low import cost |
| Datadog | Joining error records with wider observability practice | The monitoring design still needs an explicit import-success signal | Use a common identifier and meter the workflow independently from incident volume |
| Healthchecks-style monitoring | Detecting a missed scheduled check-in | It does not replace stack traces or application context | Attribute the missed run to the job and source, then link supporting error records |
| A plain REST error capability | Sending error evidence without installing a provider SDK | It is still not a heartbeat or alerting rule engine | Preserve the internal event contract and assemble the missing checks separately |
That last row is where Infrai can fit. Infrai's one REST API is pure HTTP: no SDK to install, and any language or runtime can send the request. Its API is also self-describing, so the discovery surface can show the request and response contract before a worker is wired to it; that reduces the friction of replacing an SDK-backed capture edge during a provider change. The internal event contract can remain stable while the service behind the capability changes, so swapping the provider does not require changing application code. Infrai also uses one key and one bill across the platform, which reduces credential and invoice sprawl without changing the observability limits. The platform has 295 routes across 20 modules under one key, but that breadth does not turn error capture into a heartbeat monitor or a notification system.
The 2026-10-29 capability snapshot also makes the limitations material. Infrai has no built-in threshold notifications, distributed span-tree query, source-map deobfuscation, session replay, or heartbeat monitoring in the stated capability boundary. Its client-side checking model is polling-based, and the observability facts do not establish a declared filter contract for log search or metric query. Choose Sentry when SDK-centered exception context is the priority, Datadog when a wider observability estate matters more, and a Healthchecks-style tool when missed schedules are the primary failure. Choose the REST capability only when a provider-neutral HTTP contract is more valuable than an integrated debugging suite; it isn't a replacement for one.
How should I integrate the Node.js library?
Install the selected library with npm and initialize it before application work begins. Configure middleware for uncaught exceptions and HTTP errors, then wrap the import worker as well; middleware catches request failures, while a scheduled worker may run outside an HTTP request entirely. Attach user information only where it is appropriate and lawful, and prefer a stable internal identifier over an email address.
The test should be repeatable. Trigger one controlled exception and verify that the report includes the import name and run_id. Run a successful import and verify the completion event. Then suppress the completion event in a test environment and confirm that the heartbeat check marks the run as missing. A green error dashboard with no missing-run test does not establish that the integration works.
The provider-specific SDK is an implementation detail. The durable contract is the useful part: error reports include the run context, successful work emits a completion record, and retries use the same idempotent run identifier. Do not use a trace tree as a substitute for this contract; the available logs may carry trace_id and span_id without providing distributed trace exploration.
A rollout rule that survives provider changes
Start with one import whose business owner can define “late.” Record the expected schedule, allowed delay, source, and cost center before enabling notifications. Compare observed exceptions, completed runs, and expected runs over a fixed test window. The gaps are the product decision.
Keep error capture, completion recording, and missed-run detection as replaceable edges around one internal event contract. This permits a move from Sentry to Rollbar, Datadog, or a plain REST capability without rewriting the import worker's business logic, while keeping the uncomfortable truth visible: an error tracker can explain a failed run, but only a positive completion signal can establish that a run occurred.
Top comments (0)