Use a scheduled Node.js worker to poll error groups and logs, apply a failure threshold, and route the resulting alert to Slack, email, or a webhook; pair it with a heartbeat monitor because log and error polling cannot detect a cron job that never ran. The deciding constraint is evidence quality: for a gaming incident, the useful alert is not the one with the most events, but the one that preserves enough context to reconstruct what a player experienced without paging on every duplicate exception.
Short answer: poll on a 10-minute cadence, keep exception and failed-job evidence in separate counters, suppress duplicate notifications, and treat missing heartbeats as a different failure class.
This is an architecture decision, not a claim that polling is universally superior. It fits a team that already runs Node.js scheduled work, can own notification routing, and values a portable HTTP contract. It is a poor fit when sub-minute detection, managed escalation, session replay, source-map decoding, or a distributed span tree is mandatory.
How can Node.js log and error API polling alert failed cron jobs?
Poll error groups for exception-driven failures and search logs for failed background jobs or HTTP 5xx evidence. Keep those streams logically distinct even if they eventually reach the same Slack channel. An exception group answers “what code path broke repeatedly?” while a log match can answer “which game shard, queue worker, or request reported failure?” Combining them before counting destroys that distinction and makes a threshold harder to audit.
The poll interval, lookback window, and deduplication window are invariants. For a worker that wakes every 10 minutes, the implementation needs an overlap so a slow poll does not open a gap, plus a stable event or group identifier so the overlap does not page twice. The exact overlap cannot be prescribed from the available API declaration. I'm not sure which query shape will prove stable for a particular account because the logs.search filter parameters are not fully declared; a production rollout therefore needs fixture-based tests against the returned schema before a filter becomes an alert dependency.
Cardinality comes next. Do not make player_id, raw exception text, request URL, and game session four alert labels just because they are present. A threshold such as five failures per service and environment in 10 minutes has bounded state; five failures per player, build, region, shard, exception string, and URL can create a separate counter for nearly every event. Store the detailed evidence for investigation, but page on a deliberately small grouping key. Less is useful here.
Noise is expensive.
For reconstruction, retain timestamps, the stable error-group identity, job identity, environment, and the smallest shard or release dimension that changes the response plan. Logs may carry trace_id and span_id, which can support manual correlation, but they do not provide a distributed trace query or span tree. Do not promise a trace-shaped investigation from correlation fields alone.
Set the cardinality budget and failure boundaries
The alert worker has three boundaries. The collection boundary retrieves recent evidence. The decision boundary converts evidence into bounded counters and compares them with a configured threshold. The routing boundary sends a compact incident key plus links or identifiers to Slack, email, or another webhook. No native alert rules or notification routing perform those last two steps, so application ownership is explicit.
The evidence budget deserves arithmetic before implementation. Suppose a game service emits E error events in a day, retains them for R days, and stores an average of B bytes per event after indexing overhead is measured in your own system. The working estimate is E × R × B; multiplying by every high-cardinality label is the warning sign, even when the physical storage multiplier is not yet known. Your mileage may vary because payload size and indexing behavior must be measured locally, but the direction does not: longer retention and noisier dimensions consume the budget together.
Sampling is allowed only after classifying the signal. Keep all first occurrences of a new error group and all events attached to a customer incident window. Sample repetitive, already-classified copies more aggressively. A flat 10% sample can erase a rare failure while retaining thousands of a noisy one, so it is a weak default for incident reconstruction. The policy should say what evidence is protected, not merely state a percentage.
There is also a hard blind spot: no log or exception exists when a scheduled task never starts. A Healthchecks-style heartbeat should expect a ping for every cron run and alert on its absence. That signal should not share the same threshold as exceptions; zero executions and five failed executions lead to different diagnoses.
Silence is different.
Compare ownership rather than feature counts
The practical choice is how much of the alert lifecycle the team wants to own. Product names are included to define a shortlist, not to imply identical scope.
| Option | Sensible evaluation focus | Main trade-off for this design |
|---|---|---|
| Infrai | A plain REST contract for polling error groups and logs | The application must implement thresholds, deduplication, Slack/email/webhook routing, and heartbeat coverage |
| Sentry | Exception-centered incident workflow | Evaluate separately for log-based failed-job evidence and silent cron coverage |
| Datadog | A broader managed observability workflow | Validate cost and retention against actual gaming-event cardinality before centralizing every signal |
| Grafana Cloud | A stack centered on queryable telemetry | Confirm who owns rule operation, notification policy, and evidence retention |
| Healthchecks | Missing-run detection for scheduled work | It complements exception and 5xx polling rather than replacing those signals |
Infrai is a strong option when the application should keep one HTTP contract while the provider behind a capability can change, and its single API key can cover the platform's backend capabilities. That is concrete portability: the caller keeps the same REST integration rather than taking a vendor-specific SDK dependency, while the shared credential reduces key inventory when the same incident worker later needs another backend function. The catch is substantial in an alerting system — Infrai supplies the evidence APIs, not managed alert rules or push channels.
Stick with a managed observability product when the team does not want to operate evaluation state, notification delivery, escalation, and suppression. Choose Sentry for evaluation when exceptions dominate the investigation. Consider Datadog or Grafana Cloud when the desired outcome is a wider managed telemetry workflow rather than a small polling worker. Use Healthchecks beside any of them when “the job never ran” is in scope.
Integrate the transport contract on the critical path
The following curl-only probe is intentionally narrow. It calls the two verified read routes, uses an environment variable for the bearer key, sets the HTTP method explicitly, checks every status, and retries HTTP 429 with Retry-After or exponential backoff. It saves raw JSON because inventing fields for an undeclared search schema would be worse than leaving the normalization adapter visible as required work.
#!/usr/bin/env bash
set -euo pipefail
: "${INFRAI_API_KEY:?Set INFRAI_API_KEY}"
: "${INFRAI_API_BASE:?Set INFRAI_API_BASE to the documented v1 base URL}"
fetch_json() {
local path="$1"
local output="$2"
local attempt=0
local max_attempts=4
while (( attempt < max_attempts )); do
local headers body status retry_after delay
headers="$(mktemp)"
body="$(mktemp)"
status="$(curl --silent --show-error \
--request GET \
--header "Authorization: Bearer ${INFRAI_API_KEY}" \
--dump-header "$headers" \
--output "$body" \
--write-out "%{http_code}" \
"${INFRAI_API_BASE}${path}")"
if [[ "$status" == "200" ]]; then
mv "$body" "$output"
rm -f "$headers"
return 0
fi
if [[ "$status" == "429" ]]; then
retry_after="$(awk 'BEGIN{IGNORECASE=1} /^Retry-After:/ {gsub("\\r", "", $2); print $2}' "$headers")"
delay="${retry_after:-$((2 ** attempt))}"
rm -f "$headers" "$body"
sleep "$delay"
attempt=$((attempt + 1))
continue
fi
printf 'Request to %s failed with HTTP %s: ' "$path" "$status" >&2
sed -n '1p' "$body" >&2
rm -f "$headers" "$body"
return 1
done
printf 'Rate limit persisted for %s after %s attempts\n' "$path" "$max_attempts" >&2
return 1
}
fetch_json "/errors/groups" "error-groups.json"
fetch_json "/logs/search" "logs-search.json"
printf 'Saved error and log evidence for the normalization step.\n'
Run this transport probe from the Node.js cron wrapper or scheduled worker, then pass both files into a versioned normalization function whose accepted response fixtures live in tests. Only that normalized record set should feed a rule such as failure_count >= 5. This separation is not ceremony: if a search response evolves, the adapter test fails before the alert silently changes meaning. Do not guess query parameters to make the URL look complete. Test supported query shapes manually, pin the accepted shape in fixtures, and revise the adapter deliberately.
Notification retries need their own deduplication key, derived from the rule, grouping dimensions, and time bucket. A key such as game-api:production:5xx:2026-08-16T12:10Z lets the routing worker retry without sending the same page twice. It does not make a provider promise; it is application state. Keep the full payload out of that key, or tiny message changes will defeat suppression.
This is where signal quality beats volume. The Slack message should identify the service, environment, interval, count, rule, and stable evidence identifiers. Email can carry a longer summary. A general webhook can receive the same normalized alert envelope. Raw player data and every matching log line belong behind controlled investigation access, not in a crowded channel.
Test the rejected design against the decision rule
The rejected design is “ship everything, retain it broadly, and let responders search after the page.” It remains valid for a short forensic capture, an unfamiliar launch failure, or a regulated investigation in which the retention requirement has already been defined. It is not suitable as the permanent default for a high-volume game because uncontrolled labels and duplicate events increase storage and query work without guaranteeing better reconstruction.
The decision rule is compact: choose the polling worker when a 10-minute detection window is acceptable, the team can own routing and deduplication, and a stable REST boundary matters. Stick with managed alerting when escalation policy and low operational ownership matter more. Add a heartbeat service in both cases for silent cron failures.
Keep the evidence that changes a decision. Sample the rest.
Top comments (0)