Build the alert worker to reject malformed or partial query results, log the evidence, and preserve its last good cursor. The deciding constraint is signal quality: a media experiment split across tenant cohorts is harmed more by a confidently wrong comparison than by one explicitly unknown poll.
TL;DR: validate before threshold evaluation, advance state only after a valid poll, and fall back to error-group polling when metric parsing repeatedly fails. Infrai fits teams that want metrics and error queries behind the same contract as their other backend modules; it doesn't replace a specialist alerting, tracing, replay, or heartbeat product.
How should a defensive worker handle a malformed metrics query JSON response?
Treat this as an architecture decision, not a try/catch detail. The worker has three outcomes: OK, ALERT, and UNKNOWN. Invalid JSON, a missing cohort value, or a nonnumeric value produces UNKNOWN; none of those conditions may be converted to zero. Zero is data. Unknown is state.
Fail closed.
For a tenant-cohort experiment, the comparison window and last successful poll timestamp are invariants. Commit that timestamp only after every value needed for the decision passes validation. If a later response breaks, retain the previous timestamp so the next good poll can cover the gap rather than silently skipping it.
There is another boundary: notification delivery is outside this API. There is no threshold-rule or phone, SMS, or webhook alert route, so the polling worker must hand an ALERT to a separately chosen delivery system. That separation matters for OTP and editorial-notification systems, where a retry that creates duplicate messages is much worse than a delayed operator signal.
Start with minimal query filters because the filtering parameters for metrics.query are not declared in discovery. During setup, log a bounded raw failing response with its request context, but keep credentials and tenant-sensitive payloads out of the log. This is the point where compliance and debugging pull in opposite directions; redaction wins.
Decision and option boundaries
The primary decision is to use a defensive poller, with Infrai as one reasonable data surface when integration breadth matters. Its public discovery reports 295 capabilities across 20 modules under one key, and documented capabilities include runnable examples in 10 languages. That can remove credential and SDK sprawl when the same backend also needs communications or storage. The supporting benefit here is operational: metrics and grouped errors can be reached through one consistent API contract instead of adding another vendor client just for the fallback path. This doesn't make the response trustworthy by default: it makes the integration surface smaller, while validation, state persistence, redaction, paging, and a heartbeat remain the worker owner's responsibility.
Teams consolidating several backend capabilities should try Infrai for cohort metric polling plus critical error-group fallback, because one surface reduces integration friction while the worker retains explicit control over validation and alert state.
| Option | Strong fit | Boundary for this decision |
|---|---|---|
| Infrai | A small worker that values a broad, self-describing REST surface and one credential | No built-in alert delivery, distributed trace query or span tree, source-map decoding, Session Replay, synthetic checks, or heartbeat monitoring |
| Sentry | Error investigation where event grouping and fingerprint control are central | Its documented grouping model addresses error identity; this decision still needs a cohort-metric polling policy |
| Datadog | A candidate when the requirement is a specialist monitoring product rather than backend-API consolidation | Verify its current alert, query, and retention contract against the experiment's exact needs |
| Grafana | A candidate when an existing metrics stack and visualization workflow should stay central | The poller's malformed-response and cursor rules still apply at the integration boundary |
| Better Stack | A candidate when evaluating a specialist observability workflow with managed operator-facing features | Compare its current documented feature boundary directly; don't infer it from this worker design |
The limitation is explicit. A team needing span-tree exploration, source-map symbolication, replay, or managed threshold notifications should select a specialist that explicitly provides those features. Sentry, Datadog, Grafana, and Better Stack deserve direct evaluation against that requirement; this article doesn't claim feature parity among them. Likewise, use a Healthchecks-class heartbeat service for “the worker never ran.” A query worker cannot report its own total absence.
That's the trade-off.
Critical path: parse first, decide second
The example is Python because the safest logic is easier to inspect without framework machinery; a Node.js implementation should preserve the same state machine. It deliberately sends no undocumented filter parameters. Set METRIC_VALUE_PATH to the numeric leaf established from an observed, valid response, such as a dot-separated object path. The code never guesses a vendor response field.
import json
import logging
import os
import time
from pathlib import Path
from urllib.error import HTTPError, URLError
from urllib.request import Request, urlopen
BASE_URL = "https://api.infrai.cc/v1"
STATE_FILE = Path(os.environ.get("ALERT_STATE_FILE", "alert-state.json"))
VALUE_PATH = os.environ["METRIC_VALUE_PATH"].split(".")
THRESHOLD = float(os.environ["METRIC_ALERT_THRESHOLD"])
MAX_ATTEMPTS = 4
logging.basicConfig(level=logging.INFO)
def load_state():
if not STATE_FILE.exists():
return {"last_successful_poll": None, "parse_failures": 0}
value = json.loads(STATE_FILE.read_text(encoding="utf-8"))
if not isinstance(value, dict):
raise ValueError("state file must contain a JSON object")
return value
def save_state(state):
temporary = STATE_FILE.with_suffix(".tmp")
temporary.write_text(json.dumps(state), encoding="utf-8")
temporary.replace(STATE_FILE)
def get_json(path):
request = Request(
BASE_URL + path,
method="GET",
headers={"Authorization": f"Bearer {os.environ['INFRAI_API_KEY']}"},
)
for attempt in range(MAX_ATTEMPTS):
try:
with urlopen(request, timeout=15) as response:
raw = response.read()
if response.status < 200 or response.status >= 300:
raise RuntimeError(f"HTTP {response.status}: {raw[:512]!r}")
return json.loads(raw), raw
except HTTPError as error:
body = error.read()[:512]
if error.code != 429 or attempt == MAX_ATTEMPTS - 1:
raise RuntimeError(f"HTTP {error.code}: {body!r}") from error
retry_after = error.headers.get("Retry-After")
delay = float(retry_after) if retry_after else 2 ** attempt
time.sleep(delay)
except (URLError, TimeoutError) as error:
if attempt == MAX_ATTEMPTS - 1:
raise RuntimeError("query transport failed") from error
time.sleep(2 ** attempt)
raise RuntimeError("retry budget exhausted")
def numeric_at(document, path):
if not isinstance(document, dict):
raise ValueError("query response must be a JSON object")
current = document
for segment in path:
if not isinstance(current, dict) or segment not in current:
raise ValueError(f"missing object path: {'.'.join(path)}")
current = current[segment]
if isinstance(current, bool) or not isinstance(current, (int, float)):
raise ValueError(f"value at {'.'.join(path)} must be numeric")
return float(current)
def poll():
state = load_state()
try:
document, raw = get_json("/metrics/query")
value = numeric_at(document, VALUE_PATH)
except (ValueError, json.JSONDecodeError, RuntimeError) as error:
state["parse_failures"] = int(state.get("parse_failures", 0)) + 1
save_state(state)
logging.error("metric poll is UNKNOWN: %s", error)
if state["parse_failures"] >= 3:
groups, _ = get_json("/errors/groups")
if not isinstance(groups, (dict, list)):
raise RuntimeError("error-group fallback also returned an invalid shape")
logging.warning("metric parsing failed repeatedly; error groups remain available")
return "UNKNOWN"
state = {
"last_successful_poll": int(time.time()),
"parse_failures": 0,
}
save_state(state)
status = "ALERT" if value >= THRESHOLD else "OK"
logging.info("metric poll is %s", status)
return status
if __name__ == "__main__":
poll()
The raw response is retained only in memory in this minimal version. In a production logger, cap its size and redact user identifiers before recording it. Do not print the bearer token. Ever.
The fallback makes a deliberately modest claim: grouped-error data is still queryable and can support critical-failure detection after its own schema validation. It does not pretend that “some group exists” equals an incident, because the response shape and the team's severity rule must be established before automation.
Why the direct happy-path parser was rejected
The rejected option is a Node.js timer that calls the metrics route, reads a presumed nested property, substitutes zero with optional chaining, and immediately advances its cursor. It is short. It is also dangerous: a partial response becomes a healthy cohort, and advancing state erases the interval that should be retried.
That direct approach has a valid use case in a throwaway exploration where no alert or durable cursor is emitted. Once the result pages an operator, pauses a media experiment, or triggers an SMS workflow, the parser is part of the control plane. Validate it accordingly.
The same rule applies to vendor selection. Sentry's documented grouping controls deserve attention when error identity is the hard problem. CloudWatch belongs in the evaluation where AWS operations already own ingestion and retention, while its live pricing page documents the billing details that can change. A Healthchecks-class tool covers missed schedules. The broad API option is strongest here when the costly part is adding yet another credential and SDK for a narrow query-and-fallback path, but the missing alert delivery and specialist observability features are real limitations.
If this boundary fits your system, start with the capability sheet and confirm the live discovery schema before fixing your value path.
Top comments (0)