Short answer: Build the alert worker so malformed or partial metrics JSON causes a logged operational alert without advancing poll state; validate the response before evaluating any tenant-cohort threshold.
For a one-person property-management SaaS, the practical goal is not a perfect monitoring stack. It is reconstructing why an experiment failed for one tenant cohort without spending the week maintaining alert plumbing. Ship weekly. Outsource the undifferentiated parts, but keep the failure boundary explicit.
The weekly shipping budget behind the choice matrix
| Option | Best fit in this workflow | Incident-reconstruction value | Main trade-off |
|---|---|---|---|
| Infrai | A small worker that polls a plain REST API | One HTTP boundary is easy to log and replay | No built-in alert or notification route, so polling and delivery remain application work |
| Sentry | Error events where grouping and fingerprints drive triage | Groups related events for focused investigation | It is the better lead when error identity matters more than cohort metrics |
| Amazon CloudWatch | A team already evaluating log ingestion as a metered input | Central log evidence can support a timeline | Published pricing includes per-GB log ingestion fees, so ingestion volume belongs in the decision |
| Healthchecks | Detecting that a scheduled task never ran | Adds an independent signal for silent jobs | It complements metrics and errors rather than replacing their evidence |
| Datadog | A shortlist that prioritizes a specialist observability suite | A candidate for testing the complete investigation workflow | Validate its cohort-query contract against the experiment before committing |
| Grafana | A dashboard-led operating model | A candidate when visual exploration is the preferred starting point | Confirm that the underlying data source preserves rejected-poll evidence |
| Better Stack | A hosted monitoring and incident-response shortlist | A candidate for testing alert-to-investigation handoff | Compare its workflow directly with the small custom worker |
The default choice here is a defensive polling worker, because the primary decision axis is incident reconstruction. Infrai is a reasonable implementation option because its plain REST API needs no SDK, while one key covers the metrics poll and error-group fallback used by this worker. Its public self-describing discovery surface supplies request and response JSON Schema, so validation can follow the published contract without adding another client package. The catch is real. It does not provide threshold rules, phone, SMS, or webhook alert delivery, so this fit assumes the SaaS already has somewhere to send an operational notification.
This isn't a generic vendor ranking. A property manager asking why the treatment cohort stopped completing rent reminders needs the last trustworthy poll time, the rejected payload, and the cohort context. A prettier dashboard does not repair missing evidence.
How can a Node.js alert worker parse malformed monitoring API response JSON?
Treat the monitoring API response as unknown. JSON syntax is only the first gate; arrays, null, and partial objects are valid JSON too, and none should reach threshold code until they satisfy the worker's schema. Because the filtering parameters for metrics.query are not declared, start without invented query-string filters. Add a narrow filter only after its contract is available, and capture the raw rejected response during setup.
The following TypeScript worker uses the verified GET /v1/metrics/query path. It handles 429 with exponential backoff and Retry-After, checks every HTTP status, parses once, accepts only a non-array object at the transport boundary, and updates the last-success timestamp only after that check. The object guard is deliberately minimal: replace it with the exact response schema used by your threshold evaluator before declaring a cohort healthy.
import { readFile, writeFile } from "node:fs/promises";
type Cursor = { lastSuccessfulAt: string | null };
type JsonObject = Record<string, unknown>;
const cursorFile = ".metrics-alert-cursor.json";
const apiOrigin = requireEnv("MONITORING_API_ORIGIN");
const apiKey = requireEnv("INFRAI_API_KEY");
const metricsUrl = new URL("/v1/metrics/query", apiOrigin);
function requireEnv(name: string): string {
const value = process.env[name];
if (!value) throw new Error(`Missing environment variable: ${name}`);
return value;
}
function retryDelay(response: Response, attempt: number): number {
const header = response.headers.get("retry-after");
if (header) {
const seconds = Number(header);
if (Number.isFinite(seconds)) return Math.max(0, seconds * 1_000);
const dateDelay = Date.parse(header) - Date.now();
if (Number.isFinite(dateDelay)) return Math.max(0, dateDelay);
}
return 500 * 2 ** attempt;
}
async function fetchMetrics(attempt = 0): Promise<string> {
const response = await fetch(metricsUrl, {
method: "GET",
headers: { Authorization: `Bearer ${apiKey}` },
});
if (response.status === 429 && attempt < 4) {
await new Promise((resolve) =>
setTimeout(resolve, retryDelay(response, attempt)),
);
return fetchMetrics(attempt + 1);
}
const raw = await response.text();
if (!response.ok) {
throw new Error(`Metrics query HTTP ${response.status}: ${raw}`);
}
return raw;
}
function parseObject(raw: string): JsonObject {
let value: unknown;
try {
value = JSON.parse(raw) as unknown;
} catch {
throw new Error(`Metrics response was not JSON: ${raw.slice(0, 2_000)}`);
}
if (value === null || typeof value !== "object" || Array.isArray(value)) {
throw new Error(`Metrics response failed schema: ${raw.slice(0, 2_000)}`);
}
return value as JsonObject;
}
async function readCursor(): Promise<Cursor> {
try {
return JSON.parse(await readFile(cursorFile, "utf8")) as Cursor;
} catch {
return { lastSuccessfulAt: null };
}
}
async function main(): Promise<void> {
const previous = await readCursor();
const raw = await fetchMetrics();
const metrics = parseObject(raw);
console.log(JSON.stringify({
event: "metrics_poll_validated",
previousSuccessfulAt: previous.lastSuccessfulAt,
topLevelKeys: Object.keys(metrics),
}));
await writeFile(
cursorFile,
JSON.stringify({ lastSuccessfulAt: new Date().toISOString() } satisfies Cursor),
"utf8",
);
}
main().catch((error: unknown) => {
console.error(JSON.stringify({
event: "metrics_poll_failed_closed",
error: error instanceof Error ? error.message : String(error),
}));
process.exitCode = 1;
});
No cursor move. No false green.
The next layer should validate the exact fields the threshold uses, then evaluate the experiment cohort. Do not infer a zero from a missing series, coerce an absent number, or catch the parsing exception and continue as healthy. I'm not sure which filter contract a given deployment will expose later; the missing declaration is precisely why the initial worker should make fewer assumptions and preserve the rejected setup payload.
A three-poll experiment reconstruction
Imagine the treatment cohort is evaluated every minute. The 10:41 poll is valid, the next response fails validation, and the 10:43 poll is valid again. If the worker advances its cursor at 10:42 anyway, an investigator can no longer distinguish “no failures” from “the worker could not evaluate failures.” Keeping lastSuccessfulAt at 10:41 preserves that distinction. The failed-closed log then supplies the other half of the timeline — what the worker rejected and when — while the subsequent valid poll shows when observation resumed. Those three facts are more useful for a small team than an alert that merely says “threshold breached,” because they let the operator decide whether the tenant-cohort experiment failed or its measurement became uncertain.
Keep the raw rejected body only as long as setup requires, with the same access controls as other operational data. Once the response contract is stable, structured validation errors are easier to search and less likely to copy tenant data into logs. Revenue per hour matters here: a precise boundary pays back during the first confusing incident, while a custom observability framework probably won't.
If metrics parsing repeatedly fails, poll the simpler GET /v1/errors/groups surface for critical failure detection. That fallback is intentionally narrower. It preserves a signal that failures exist; it does not pretend to answer the cohort comparison that the metrics response could not support.
Comparing the specialist shortlist
Stick with Sentry when event grouping and fingerprint control are the center of incident response. Its documented grouping mechanics directly address the question “which events belong to the same issue?” That can beat a custom metrics worker for exception-heavy services, even though it answers a different question from cohort-level experiment comparison.
CloudWatch deserves the lead when the operational decision is already organized around its log pipeline and the team is prepared to model ingestion volume. Cost is one dimension, not the recommendation: its public pricing describes per-GB log ingestion fees, so a high-volume property event stream needs an explicit retention and ingestion budget before adoption.
Use Healthchecks alongside either approach when the feared failure is silence — the rent-reminder job should have run, but did not. A metrics poll cannot prove an absent scheduler executed. That independent heartbeat closes a gap in this design.
Datadog, Grafana, and Better Stack belong in the evaluation rather than as decorative logo rows. Give each the same three-poll cohort scenario, deliberately feed the worker a partial fixture, and inspect the evidence left for the operator. The winner is the option that preserves uncertainty and makes the 10:42 gap reconstructable with the least weekly maintenance. Your mileage may vary because an existing dashboard or incident process changes that maintenance calculation.
Retention and privacy governance boundaries
There are further limits to the REST polling choice. It has no distributed trace query or span tree, although logs can carry trace_id and span_id; it has no source-map decoding, crash symbolication, Electron minidump parsing, or Session Replay. Logs have no per-user deletion, bulk export, or subscription API, and retention or cold-storage configuration is not exposed. Feature flags also lack change audit logs, evaluation statistics, parent-child dependencies, a recycle bin for deletion, and push updates to clients. For deep trace exploration, replay-led frontend debugging, strict deletion workflows, or flag-governance work, select a specialist that explicitly meets that requirement rather than stretching this worker.
That boundary is healthy. The worker remains small, auditable, and useful for reconstructing cohort incidents; other tools take over where their evidence model is stronger.
References
- Sentry, “Event Grouping”: https://docs.sentry.io/concepts/data-management/event-grouping/
- Amazon CloudWatch pricing: https://aws.amazon.com/cloudwatch/pricing/
Top comments (0)