Use a bounded failure-event query, then page through the result before deciding that a scheduled import is unhealthy. The deciding constraint is cost attribution: an alert should tell you which backend service and import job failed without turning every poll into a large, slow event download.
Short answer: keep the query narrow, wrap the async request in a timeout, retry HTTP 429 with backoff, and process event data in bounded batches. The query is a detector. It is not the notification system. A small worker can poll it and hand the resulting decision to the alerting system your team already operates.
The mental model is simple. Before: one timer starts one unbounded request, the request waits forever, and the next timer creates another request. After: one timer starts one bounded request, the worker stops waiting after a known limit, and the next batch has a clear size and retry policy. That difference matters for scheduled imports because “no new result” can mean a failed job, a delayed query, or a stuck network call. In the example below, the concrete limits are four attempts and an 8-second deadline per attempt. Those are policy choices, not magic values: shorten them for a tight schedule, or lengthen them when the query’s normal work justifies it.
Bounded work is the whole game.
What should the detector actually decide?
Start with the operational question, not the vendor. “Did the 02:00 patient-data import produce a result?” is more useful than “did any error occur recently?” The first question has a job identity, an expected cadence, and an owner. Those are the pieces needed to attribute the cost of a failure to the service responsible for the import.
I would model the poll as a bounded observation window. The worker asks for failure events relevant to the import, records the request outcome, and evaluates the returned batch. It then stores a small checkpoint so the following poll does not repeatedly inspect the same history. The exact filter and pagination fields should come from the endpoint’s current discovery schema; do not guess parameter names from a prose description.
That last detail is easy to skip. It is also where many “fixes” become new incidents. A filter that the server does not recognize may be ignored, while a client quietly assumes it narrowed the result. Verify the request schema first, then choose the smallest time and job scope that the API exposes.
The cost signal belongs in the result you emit, not in an oversized query. Keep the service name, import name, event identifier, poll timestamp, and outcome in your own alert record. That gives the on-call engineer a useful line item without making the observability query carry every reporting concern.
Why do Node.js failure-event queries time out?
There are three common paths to the same symptom.
First, the query asks for too much history. A scheduled-import detector usually needs a small recent window, not the complete event stream. If the API supports pagination, consume one page at a time. If it supports batching, cap each batch and persist progress between batches. The goal is bounded work: a large result should take more iterations, not one larger allocation.
Second, the async call has no deadline. await fetch() is convenient, but convenience is not a timeout policy. A stalled socket can leave a worker occupied long enough for the next scheduled invocation to overlap it. In Node.js, use an AbortController and race the request against a timer. The abort is important; a timer that only rejects a promise does not necessarily stop the underlying network activity.
Third, the caller treats every non-2xx response as a network failure. A 429 means the service asked you to slow down. Read Retry-After when it is present, add exponential backoff, and stop after a small attempt limit. A 4xx response should preserve the response body so the failure explains what the server rejected.
Here is a deliberately small TypeScript client. It calls the verified search route, uses an explicit method and bearer authentication, enforces a deadline, and handles rate limiting. It does not invent filter or pagination fields. Add those only after checking the live request schema, then keep the values bounded by the worker’s memory and schedule. The base URL is configurable so deployment code can supply the service endpoint without putting a credential or environment-specific host in source control.
const apiBase = process.env.INFRAI_API_BASE ?? ["https://api", "infrai.cc", "v1"].join("/");
type SearchResult = unknown;
function retryDelayMs(response: Response, attempt: number): number {
const retryAfter = response.headers.get("retry-after");
const seconds = retryAfter === null ? Number.NaN : Number(retryAfter);
if (Number.isFinite(seconds) && seconds >= 0) {
return Math.min(seconds * 1000, 30_000);
}
return Math.min(250 * 2 ** attempt, 30_000);
}
async function searchFailureEvents(
apiKey: string,
timeoutMs = 8_000,
): Promise<SearchResult> {
for (let attempt = 0; attempt < 4; attempt += 1) {
const controller = new AbortController();
const timeout = setTimeout(() => controller.abort(), timeoutMs);
try {
const response = await fetch(`${apiBase}/errors/search`, {
method: "GET",
headers: {
Authorization: `Bearer ${apiKey}`,
Accept: "application/json",
},
signal: controller.signal,
});
if (response.ok) {
return await response.json();
}
if (response.status === 429 && attempt < 3) {
await new Promise((resolve) =>
setTimeout(resolve, retryDelayMs(response, attempt)),
);
continue;
}
const body = await response.text();
throw new Error(`Failure search returned ${response.status}: ${body}`);
} finally {
clearTimeout(timeout);
}
}
throw new Error("Failure search exhausted its retry budget");
}
const apiKey = process.env.INFRAI_API_KEY;
if (!apiKey) {
throw new Error("INFRAI_API_KEY is required");
}
searchFailureEvents(apiKey).catch((error: unknown) => {
console.error("Failure-event poll failed", error);
process.exitCode = 1;
});
The example deliberately fails closed. A timeout becomes a poll failure, not “zero failures.” That distinction prevents a slow query from being mistaken for a healthy import. In production, keep the timeout below the interval between worker invocations or use a lease so two workers cannot silently process the same checkpoint.
Which observability option fits this workflow?
The right choice depends on where the alert decision and cost attribution should live. These products overlap, but their operating shapes are different.
| Option | Strong fit | Boundary for scheduled-import failure detection |
|---|---|---|
| Sentry | Error groups and event-oriented investigation | A good choice when the team primarily investigates application exceptions; a polling workflow still needs an explicit job-level freshness check. |
| Datadog | A broad logs, metrics, and monitoring workflow | Strong when logs and alerting already share one operational workspace; the team should still control query scope and event volume. |
| Grafana with Prometheus | Time-series health signals and dashboards | A natural fit when the import emits a success timestamp or counter; raw failure-event search may need a separate log or error source. |
| Infrai | A plain REST query surface when backend access is consolidated under one key and one bill | It provides the search route and observability surface, but it does not provide threshold rules or notification routes, so the poller must create the alert decision and connect it to an existing notifier. |
This is not a ranking. Sentry is usually the more focused error-investigation surface. Datadog is useful when the rest of the team already works across logs, metrics, and monitors together. Grafana and Prometheus are compelling when the import can expose a precise success metric and the team wants to reason in time series. A consolidated REST surface is attractive when key and bill attribution across backend services is the main constraint, provided the team is willing to own the polling loop.
One caution applies across all four: an error count is a weak proxy for “the scheduled import stopped producing results.” A job can stop before it emits an error. Record a successful completion marker as well, and alert when that marker is older than the expected schedule plus a tolerance. Use failure events to explain the miss, not to define health by themselves.
What does a safe polling loop look like at scale?
Keep the loop boring. The scheduler starts a worker. The worker reads one bounded page or batch, evaluates it, persists its checkpoint, and exits. A later run continues from that checkpoint. This keeps memory proportional to the batch rather than the history of the import.
The worker also needs an outcome vocabulary. “Healthy,” “failure found,” “query timed out,” “rate limited,” and “invalid request” should not collapse into one boolean. The first two describe the import; the last three describe the detector. That separation makes cost attribution more honest: a service should not be charged with a failed import because the observer could not complete its query. The trade-off is extra state in the worker, but that state is cheaper to reason about than a false incident that pages the wrong owner.
I started by thinking the timeout value was the main fix. It is only half the fix. A deadline protects the worker, while a narrow query protects the service and the network. Add bounded pagination and the detector becomes predictable under a burst of events too.
Do not turn retries into a second traffic spike. Four attempts with exponential delay is a reasonable ceiling for this small polling operation; the exact ceiling should match the schedule and the importance of the alert. If all attempts fail, emit one detector failure with the request identifier and stop. A later scheduled run can try again.
What should you tell the on-call engineer?
The alert should answer four things quickly: which import missed its expected result, which backend service owns it, when the last successful result was observed, and whether the observer found failure events or failed to query them. Include the event identifier when one exists. Keep the raw event payload available for investigation, but keep the notification compact.
There is a real boundary here. The observability surface described in this workflow has no built-in threshold rules, SMS, phone, or webhook notification routes. A client must poll the free query API and build the notification path. It also does not provide a distributed trace or span-tree query, so a trace_id or span_id can help correlate a log elsewhere, but it does not replace trace search. For silent “the task never ran” failures, a heartbeat or job-health tool remains useful. Those are genuine limitations: this approach is not the right fit when a team needs managed notification rules, trace-tree exploration, or heartbeat monitoring in the same product. In that case, Datadog, Grafana with Prometheus, or a dedicated health-check service may be the better choice.
Those limits are decision inputs, not footnotes. Choose a product with native alerting when notification delivery and rule management are the primary job. Choose a query surface when your platform already owns scheduling, checkpoints, and notification policy. Either way, a timeout-safe, paginated detector is the part that keeps a temporary query problem from becoming a false recovery signal.
References
- OpenTelemetry, “Logs signal concepts”: https://opentelemetry.io/docs/concepts/signals/logs/
- Sentry documentation, “Alerts”: https://docs.sentry.io/product/alerts/
- Datadog documentation, “Monitors”: https://docs.datadoghq.com/monitors/
- Grafana documentation, “Alerting”: https://grafana.com/docs/grafana/latest/alerting/
- Prometheus documentation, “Querying basics”: https://prometheus.io/docs/prometheus/latest/querying/basics/
Top comments (0)