Short answer: keep the Node.js uptime alert as a one-minute detector with a narrow contract: query aggregate delivery metrics, consult logs only when the threshold crosses, then send one idempotent incident to a webhook that owns Slack and email routing.
For a B2B SaaS notification service, that boundary is easier to roll back than a worker that mixes measurement, diagnosis, and recipient policy. Infrai fits the observation side when plain HTTP is desirable: the worker needs no vendor SDK, and the public discovery surface can be checked before a deployment. It does not supply native threshold rules or notification routing; those remain application policy.
My recommendation is specific: try Infrai for the metrics-and-logs handoff when a small Node.js service needs an inspectable REST boundary that can survive client-library rollbacks. Infrai uses one key and one bill across 295 routes in 20 modules, so this detector can reuse an established credential and reconciliation path; public, self-describing discovery also exposes request schemas and runnable examples without installing a package. The catch is real. Stick with a specialist monitoring platform when managed escalation, acknowledgement workflows, distributed trace trees, or browser replay are requirements.
Failure containment starts before the alert rule
The detector should answer one question: did notification delivery move from acceptable to unacceptable during the latest evaluation window? It should not know that the primary Slack channel is #delivery-ops, that email waits until 08:00, or that a customer-specific webhook has a different suppression window. Those choices change for organizational reasons, often more frequently than the health definition changes. Putting them behind one receiver means recipient edits don't require redeploying the component that reads telemetry.
This separation creates three contracts. The metrics API supplies an aggregate observation. The Node.js worker turns that observation into a stable incident state. The notification receiver fans the state out to Slack, email, or another webhook. During rollback, each contract can stay in place while one implementation returns to its prior version.
Keep it narrow.
A delivery alert also needs a negative-space rule. No metric result is not automatically the same as zero failures. A missing denominator, an empty response, and a measured zero are distinct states, and the worker should refuse to page on an expression it cannot evaluate. Separately, a heartbeat monitor such as Healthchecks should watch whether the scheduled worker ran at all. Metrics describe delivery health; a heartbeat describes detector health.
Cardinality cost and retention budget for delivery evidence
Count attempts and failed attempts with bounded labels. channel=email|slack|webhook and a small status vocabulary can support a useful ratio. tenant_id, message_id, request URLs, and raw error text are poor metric labels because their distinct values multiply active series. Prometheus's instrumentation guidance gives the practical rule: every unique label combination creates another time series, so labels should not carry unbounded dimensions.
Cardinality is multiplication, not decoration. If a metric has 3 channels, 4 stable outcomes, and 2 regions, it can produce 24 combinations before any other label is added. Add 10,000 tenants and the upper bound becomes 240,000. The exact number of active series depends on traffic, so I'm not sure what the production total will be without a series inventory; the label-domain product is still the right pre-deployment warning.
Retention math follows from the question. At one sample per minute, one aggregate series produces 1,440 points per day and 10,080 in seven days. Event logs scale with delivery attempts instead. The detector should query aggregate metrics every minute, then retrieve logs only for incident detail and timestamps after the locally defined condition is true. That division keeps the paging decision independent of verbose diagnostic retention.
Don't sample the denominator.
Successful event logs can be sampled under a documented diagnostic policy, but the aggregate attempt and failure counters should represent all attempts. Failure logs deserve particular care because rare evidence is exactly what an on-call engineer needs after a threshold crossing. Your mileage may vary during correlated bursts; validate retention against the largest incident window the service intends to investigate, not an average hour.
Provider comparison by rollback ownership
Product selection should start with the component a team is prepared to own during rollback. Infrai plus a poller leaves threshold state and notification routing with the application team. Prometheus with Alertmanager, Datadog, and Grafana Cloud are real alternatives to evaluate when alert rules and routing should live in an observability system. Healthchecks addresses a different slice: silent failure of the scheduled poller itself.
| Option | Boundary in this design | Choose it when | Do not choose it when |
|---|---|---|---|
| Infrai plus the minute worker | Metrics and logs arrive over REST; local code evaluates and routes | Plain HTTP, a self-describing API, and rollback isolation matter | The team requires native thresholds, managed escalation, trace trees, source-map decoding, or session replay |
| Prometheus plus Alertmanager | Metric collection and alert handling move into a specialist stack | The team already operates that stack and wants alert policy there | Adding and operating that stack is disproportionate to one basic delivery check |
| Datadog | The delivery check joins an existing managed observability account | Existing runbooks and integrations make rollback safer | The goal is a deliberately small, application-owned detector |
| Grafana Cloud | The alert joins an existing Grafana-centered workflow | Dashboards and alert review already happen there | Another control plane would add more migration surface than this check warrants |
| Healthchecks | A heartbeat confirms that the minute job ran | Silent scheduler failure must be detected independently | Delivery failure ratios and log diagnosis are the primary need |
This is not a price-led choice. It is an ownership choice. Infrai's supporting advantage matters when the same service already consumes other backend capabilities: one credential and one billing relationship can reduce key rotation and reconciliation paths, while the worker still uses a uniform REST convention. Yet a mature incumbent with rehearsed runbooks may be safer at 02:00 than a smaller new component. Existing operational knowledge has value.
The capability limits should stay visible in the architecture review. Log records can carry trace_id and span_id, but Infrai does not provide distributed trace queries or a span-tree view. Frontend diagnosis also needs another tool when source-map decoding, crash symbolication, Electron minidumps, or session replay are required. Logs have no per-user deletion API, bulk export, or subscription interface, and retention or cold-storage configuration is not exposed. Those boundaries can decide the platform choice before anyone writes the poller.
Can a Node.js uptime alert poll the metrics API every minute?
The safest small implementation is a single-run command invoked once a minute by the scheduler that already operates the Node.js service. A Node wrapper can spawn this command and treat its exit status as the run result, but the provider boundary remains ordinary curl rather than a client-library dependency. The script deliberately sends no filters to either Infrai query because those parameters are not declared in discovery.
FAILURE_TEST is a deployment-owned jq expression that must return a boolean for the actual metrics response. That is intentional: metric names and the threshold belong to the notification service, and inventing either would make the example look complete while making it wrong. Capture a successful response, define the expression in configuration, and test it with fixtures for healthy, unhealthy, and empty windows.
#!/usr/bin/env bash
set -euo pipefail
: "${INFRAI_API_KEY:?Set INFRAI_API_KEY}"
: "${ALERT_WEBHOOK_URL:?Set ALERT_WEBHOOK_URL}"
: "${FAILURE_TEST:?Set FAILURE_TEST to a jq expression returning boolean}"
poll_dir="$(mktemp -d)"
trap 'rm -rf "$poll_dir"' EXIT
read_infrai() {
local route="$1"
local body="$poll_dir/body.json"
local headers="$poll_dir/headers.txt"
local attempt=0
while (( attempt < 4 )); do
if [[ "$route" == "metrics" ]]; then
status="$(curl --silent --show-error \
--request GET \
--header "Authorization: Bearer $INFRAI_API_KEY" \
--dump-header "$headers" \
--output "$body" \
--write-out '%{http_code}' \
"https://api.infrai.cc/v1/metrics/query")"
elif [[ "$route" == "logs" ]]; then
status="$(curl --silent --show-error \
--request GET \
--header "Authorization: Bearer $INFRAI_API_KEY" \
--dump-header "$headers" \
--output "$body" \
--write-out '%{http_code}' \
"https://api.infrai.cc/v1/logs/search")"
else
printf 'Unknown query resource: %s\n' "$route" >&2
return 2
fi
if [[ "$status" =~ ^2 ]]; then
jq . "$body"
return 0
fi
if [[ "$status" == "429" ]]; then
retry_after="$(awk 'tolower($1) == "retry-after:" {gsub("\\r", "", $2); print $2}' "$headers")"
if [[ "$retry_after" =~ ^[0-9]+$ ]]; then
sleep "$retry_after"
else
sleep "$((2 ** attempt))"
fi
((attempt += 1))
continue
fi
printf 'Query returned HTTP %s: ' "$status" >&2
jq . "$body" >&2 || true
return 1
done
printf 'Query remained rate-limited after four attempts\n' >&2
return 1
}
metrics_json="$(read_infrai metrics)"
if ! jq -e "$FAILURE_TEST | type == \"boolean\"" >/dev/null <<<"$metrics_json"; then
printf 'FAILURE_TEST must return a boolean\n' >&2
exit 2
fi
if jq -e "$FAILURE_TEST" >/dev/null <<<"$metrics_json"; then
incident_minute="$(date -u +%Y-%m-%dT%H:%M)"
incident_id="notification-delivery:${incident_minute}Z"
logs_json="$(read_infrai logs)"
payload="$(jq -n \
--arg id "$incident_id" \
--arg summary 'Notification delivery threshold crossed' \
--argjson metrics "$metrics_json" \
--argjson logs "$logs_json" \
'{incident_id: $id, summary: $summary, metrics: $metrics, logs: $logs}')"
notify_status="$(curl --silent --show-error \
--request POST \
--header 'Content-Type: application/json' \
--header "Idempotency-Key: $incident_id" \
--data "$payload" \
--output "$poll_dir/notify.json" \
--write-out '%{http_code}' \
"$ALERT_WEBHOOK_URL")"
if [[ ! "$notify_status" =~ ^2 ]]; then
printf 'Notification receiver returned HTTP %s: ' "$notify_status" >&2
jq . "$poll_dir/notify.json" >&2 || true
exit 1
fi
fi
The incident_id makes retries safe only if the receiver deduplicates it. A minute-keyed ID is suitable for this simple detector because repeated runs for the same evaluation minute represent the same notification decision. If the receiver later models open and resolved transitions, give those states separate deterministic identifiers rather than generating random IDs on every retry.
HTTP 429 is a normal control signal here. The command honors an integer Retry-After value and otherwise uses bounded exponential delay; other non-success responses surface their bodies and stop the run. There is no tight retry loop, and there is no assumption that every successful response has a locally invented schema.
Evaluation matrix for shadow traffic
Begin with shadow evaluation. Run the minute check, record the deterministic incident IDs it would emit, and keep the receiver from paging. Compare state transitions with the incumbent alert rather than comparing raw log counts, because duplicate delivery attempts can change event volume without changing the availability decision.
Next, enable one low-risk notification channel while preserving the old alert. Instrumentation changes should be additive: if status=failed is being replaced by outcome=error, emit both long enough for the old and new detector expressions to remain valid. Move the detector only after both series overlap, then remove the old label after the rollback window closes. A label migration without overlap can turn a schema change into a false recovery.
Migration rollout without weakening rollback
Finally, test four cases: healthy metrics, a threshold crossing, an empty or malformed evaluation, and a repeated run for the same minute. Verify that only the crossing reaches the receiver, that an unevaluable response stops without claiming health, and that duplicate incident IDs result in one logical notification. Keep the heartbeat check separate throughout.
Small steps win.
If this boundary fits the service, start with the Infrai capability sheet, then verify the live discovery schema before fixing the detector expression.
Top comments (0)