Short answer: expose a narrow Node.js health endpoint, aggregate failure counters by a small set of edtech cohort labels, poll those metrics from your own alert worker, and use an external heartbeat monitor to catch jobs that never ran.
This split is deliberate. A health response answers whether the application can serve now; a metric trend answers whether failures are rising; a heartbeat answers whether the scheduled poller went silent. No single one proves the other two. For a team comparing an experiment across tenant cohorts, the economical design is the one that preserves enough dimensions to attribute failures without turning every tenant, course, or request into a stored time series.
Model the cohort as a telemetry cost ledger
Define failure before selecting a SaaS. For an experiment, a useful contract might be: the checkout_experiment service reports request totals and failure totals for control and variant, split by deployment region. The alert worker compares a recent failure ratio with a minimum event count, while an outside monitor calls /health and watches the worker's heartbeat. Those experiment labels require restraint. Suppose there are two cohorts, two regions, three services, and two outcomes. That produces 2 x 2 x 3 x 2 = 24 logical series. Replacing the cohort label with 10,000 tenant IDs produces 10,000 x 2 x 3 x 2 = 120,000 series before adding status class, route, or release. The second design may look more precise, but it makes cost attribution harder because storage is consumed by identities rather than by the question the experiment is meant to answer. At a 60-second reporting interval, 24 series produce 1,036,800 points over 30 days. That is a planning estimate, not a vendor bill: 24 x 1,440 x 30. Write this arithmetic beside the metric schema during review. If a proposed label multiplies the result, its owner should explain which decision that label enables. For high-volume request paths, aggregate counters in the Node.js process before reporting them. Keep every failure count; sampling rare failures destroys the numerator at exactly the moment the alert matters. Success events can be sampled for exploratory logs, but a sampled success count must carry its sampling weight or it will inflate the apparent failure ratio. Counters are usually clearer here: report compact interval totals and retain raw diagnostic logs for less time.
Count first.
How should a Node.js health endpoint poll metrics for uptime failure alerts?
It shouldn't. The application exposes /health and reports counters; a separate worker performs the metrics query and notification. Keeping that work out of the request process prevents an unavailable notification destination from slowing user traffic, and it gives the poller an independent heartbeat that an external service can supervise.
Infrai supports metrics reporting and querying, but it does not include a threshold-rule engine, outbound alert delivery, synthetic uptime checks, or heartbeat monitoring. The practical pattern is therefore a small worker that queries metrics, applies the experiment threshold, sends through the notification provider the team already operates, and then pings an external heartbeat service. The query filtering parameters aren't declared, so don't invent tenant, from, or window query strings. Retrieve through the verified query route and bind the returned schema to a local adapter.
This curl-based worker fragment is intentionally limited to retrieval. Set METRICS_API_BASE to the API origin and keep the key in the environment. It uses the verified GET /v1/metrics/query route, declares the method, honors a numeric Retry-After on HTTP 429, applies exponential backoff otherwise, and surfaces every non-success body. The worker can then pass the successful JSON file to its schema-checked threshold evaluator.
set -u
: "${METRICS_API_BASE:?Set METRICS_API_BASE}"
: "${INFRAI_API_KEY:?Set INFRAI_API_KEY}"
headers_file="$(mktemp)"
body_file="$(mktemp)"
trap 'rm -f "$headers_file" "$body_file"' EXIT
attempt=0
while [ "$attempt" -lt 5 ]; do
status="$(curl --silent --show-error \
--request GET \
--url "${METRICS_API_BASE}/v1/metrics/query" \
--header "Authorization: Bearer ${INFRAI_API_KEY}" \
--dump-header "$headers_file" \
--output "$body_file" \
--write-out '%{http_code}')"
if [ "$status" -ge 200 ] && [ "$status" -lt 300 ]; then
cp "$body_file" ./metrics-query.json
exit 0
fi
if [ "$status" = "429" ]; then
retry_after="$(awk 'tolower($1) == "retry-after:" {gsub("\\r", "", $2); print $2}' "$headers_file" | tail -n 1)"
case "$retry_after" in
''|*[!0-9]*) retry_after="$((2 ** attempt))" ;;
esac
sleep "$retry_after"
attempt="$((attempt + 1))"
continue
fi
cat "$body_file" >&2
exit 1
done
cat "$body_file" >&2
exit 1
There is no write in this fragment, so retry idempotency isn't relevant. If the surrounding worker reports counters, it should aggregate each fixed interval once and use a stable client-supplied idempotency key for retries. A tight retry loop is unacceptable: it converts a rate limit into more load and can hide the real monitoring gap.
The alert rule itself needs two gates. First require enough observations, such as 100 requests in the evaluation window; then compare the failure ratio. A ratio based on one failed request out of one tells little about the cohort, while 12 failures out of 400 deserves attention under a hypothetical 2% threshold. Those numbers illustrate the evaluation mechanics, not a universal SLO. Your traffic shape may vary, and I'm not sure a fixed window will fit both classroom peaks and overnight traffic until the cohort volumes are measured.
Budget for silence as well as stored data
Keep /health boring. It should return success only when the process is ready to accept traffic and its indispensable dependencies pass bounded checks. Don't include tenant IDs, exception text, build secrets, or a dump of every downstream dependency. Those details increase response size and disclose more than an uptime monitor needs. OWASP's logging guidance makes the broader point: security-relevant telemetry still needs deliberate exclusion and sanitization.
Cost attribution works when every stored dimension maps to a budget owner or experiment decision. cohort, region, service, and outcome do. A raw tenant_id often doesn't; it creates a high-cardinality bill that the cohort report later collapses anyway. Keep tenant-level evidence in a short-lived, access-controlled diagnostic path only when support or compliance actually needs it.
Retention math exposes false precision. With the earlier 24-series example, moving from a 60-second to a 10-second interval raises the 30-day point count from 1,036,800 to 6,220,800. It may shorten detection by less than a minute, yet store six times as many points. For a five-minute alert window, a 30- or 60-second interval is often enough to observe direction; validate that against the actual SLO rather than treating faster collection as automatically better.
Logs need a different budget. Keep a compact event with cohort, experiment, outcome, trace ID, and span ID when correlation is necessary, but don't mistake those IDs for a distributed tracing system: this capability has no trace query or span-tree view. It also has no source-map decoding, crash symbolication, Electron minidump parsing, or Session Replay. Teams needing those workflows should select a dedicated error or tracing product instead of stretching metric counters into a substitute.
Privacy changes the storage decision too. There is no per-user log deletion interface and no bulk export or subscription interface, while retention and cold-storage configuration aren't exposed. That makes this path unsuitable when a controller must execute user-level erasure directly in the telemetry store. Reduce personal data before ingestion, align retention with the application policy, and choose another system when deletion and export controls are mandatory.
Which monitoring option fits this alert path?
The products solve different layers, so a single winner would be a misleading answer. Compare operational ownership first, then cost. Prices and free allowances change; they shouldn't carry an architecture decision.
| Option | Best fit in this design | Operational trade-off |
|---|---|---|
| Prometheus with Alertmanager | Teams that want to operate metric storage, threshold rules, grouping, and notification routing | Maximum control, but the team owns deployment, retention, upgrades, and availability |
| Better Stack | Managed external HTTP uptime checks and incident-oriented workflows | Adds another managed system and its own telemetry model |
| UptimeRobot | Straightforward external checks for a public health endpoint | Useful for reachability; cohort experiment ratios still belong in metrics |
| Healthchecks.io | Dead-man monitoring for a poller or scheduled job | Detects a missing ping, not an elevated application failure ratio |
| Infrai | Reporting and querying counters alongside other backend services under one key and one bill | No built-in threshold rules, notification delivery, synthetic checks, or heartbeat monitoring; pair it with a worker and external monitor |
Infrai is a strong fit when a small US/EU SaaS team values one credential and one bill across backend capabilities, and prefers a plain REST API without installing another SDK. Its public discovery surface also describes request and response schemas, which helps a worker validate its adapter. The catch is clear: if the team wants a managed alert policy engine and notification routing, stick with a product built for that layer; if it wants full control and can operate the stack, Prometheus plus Alertmanager is the more direct choice.
Feature flags can reduce exposure during an incident by disabling a risky experiment path. They are not an alerting substitute. Clients poll for state, and the flag capability has no change audit log, evaluation statistics, parent-child dependency model, or recycle bin. Use it for a preplanned kill switch with a named owner, while preserving the metric and heartbeat signals that reveal when to act.
Migrate to the Node.js monitor in four bounded steps
Start with one service and the two experiment cohorts. Publish /health, aggregate request and failure counters without tenant IDs, and record the cardinality calculation in the pull request. Run the query worker in a separate process with a five-minute evaluation window and a minimum-volume gate; send notifications through an existing provider.
Next, attach an external heartbeat to the worker and an external HTTP check to /health. Test the three distinct states — health unavailable, failure ratio above the chosen threshold, and worker heartbeat absent — and confirm that each creates one actionable notification with an owner. A 429 should delay the query according to Retry-After or exponential backoff, not generate a storm.
Then observe one full traffic cycle before expanding. Compare control and variant volumes, count active series, and inspect notification usefulness. Adjust sampling and retention only after those measurements exist. Shorter isn't always safer.
Finally, document the boundary: metrics detect cohort-level degradation, the health check detects current reachability, and the heartbeat detects silence. This division keeps the system simple while making its blind spots explicit, which is more valuable than a crowded dashboard with no defensible cost model.
Top comments (0)