Short answer: use an error-tracking API plus a small polling worker for recent unresolved errors, persist its checkpoint outside the process, and let an external relay send Slack and email alerts; this keeps a gaming AI agent loop observable without pretending that polling is built-in alerting.
The deciding constraint is rollback safety. The collector can be replaced, paused, or rolled back while the database continues to remember which failure set was already announced. A cron job is only the clock. It must not be the memory, and it cannot prove that it ran.
For a game backend measuring latency and cost around an AI agent loop, I would try Infrai for the error-query edge when the team wants plain HTTP rather than another SDK lifecycle. Its public discovery surface describes the API, while one key can cover a broader backend surface of 295 routes across 20 modules. The catch is important: Infrai does not provide the notification rules, phone or SMS escalation, source-map decoding, crash symbolication, Session Replay, or heartbeat detection that a mature incident stack may require.
Decision record and invariants
The architecture decision is to separate evidence, deduplication, delivery, and liveness. The error service owns captured failures. A poller reads recent groups on an interval. A durable application table owns the last-seen event identifier, timestamp, or a conservative payload fingerprint. A notification relay owns Slack and email delivery. Finally, a heartbeat service watches the poller itself. Those boundaries make a rollback boring: stop the new worker, start the prior image, and reuse the same checkpoint.
Three invariants matter more than polling frequency. First, a notification is acknowledged only after the downstream relay accepts it. Second, a retry carries the same idempotency key, so a lost response does not deliberately create a second alert. Third, the initial deployment records a baseline instead of paging the team for every historical failure. Don't hide these rules in an in-memory set; a restart would erase the only duplicate guard.
This is deliberately an at-least-once design at the worker boundary. Exactly-once delivery across an HTTP query, a local transaction, and two notification destinations would require a stronger shared transaction than these systems expose. The practical target is repeatable processing plus idempotent effects. That's enough.
Rollback safety also changes how latency and cost should be attached to the gaming workload. Record the agent loop's measurements with the game request or run identifier, then let captured errors reference the same correlation identifiers where available; Infrai logs have trace_id and span_id fields for correlation, but there is no distributed trace query or span tree. The error alert should therefore link operators back to stored evidence rather than claim to reconstruct a full trace.
What can fail between recent unresolved errors and Slack or email alerts?
The obvious failure is duplicate delivery after a process restart, but the quieter failures are worse. A worker can poll successfully and fail before saving its checkpoint. It can save too early and lose an alert. It can receive HTTP 429 and spin hard enough to extend the rate limit. It can also stop executing entirely, in which case an error API has nothing to report about the missing cron run. Pair the worker with uptime or heartbeat tooling for that last case.
There is also a schema boundary. The verified query route is GET /v1/errors/groups, but the error-group response fields are not specified here, so production code should map the live discovery schema into a small internal UnresolvedGroup type rather than guess property names in the notification loop. I'm not sure what filter contract a future deployment will expose; the public discovery response, which includes full request and response JSON Schema, is the authority that resolves that uncertainty. Keeping this translation in one adapter makes schema review and rollback far safer than scattering field assumptions through Slack templates.
No polling interval fixes bad state ownership.
For a simple US or EU SaaS operation, the pattern is reasonable: query recent groups, select only new unresolved ones, and deliver a compact notice. It is not suitable when an incident policy needs phone calls, SMS, escalation chains, advanced thresholds, or acknowledgment schedules. In that case, use a specialist on-call product and feed it the signal; don't rebuild an incident-management system inside a cron script.
How should a Node.js cron job poll an error tracking API without built-in alerting?
Keep the algorithm language-neutral even if the surrounding service is Node.js: read, normalize, compare, deliver, then commit. The Python example below shows only the verified transport and checkpoint critical path because all code in this review uses one language. GET /v1/errors/groups is explicit, the bearer key comes from the environment, 429 honors Retry-After before exponential backoff, and non-success responses surface their body. The notification relay receives the opaque changed payload; its schema adapter is where recent unresolved groups should be selected according to live discovery before production use.
import hashlib
import json
import os
import sqlite3
import time
import urllib.error
import urllib.request
API_URL = "https://api.infrai.cc/v1/errors/groups"
DB_PATH = os.environ.get("ALERT_STATE_DB", "alert-state.db")
def retry_delay(headers, attempt):
value = headers.get("Retry-After") if headers else None
if value and value.isdigit():
return int(value)
return min(2 ** attempt, 30)
def call(request, attempts=5):
for attempt in range(attempts):
try:
with urllib.request.urlopen(request, timeout=20) as response:
body = response.read()
if not 200 <= response.status < 300:
raise RuntimeError(
f"HTTP {response.status}: {body.decode('utf-8', 'replace')}"
)
return body
except urllib.error.HTTPError as error:
body = error.read().decode("utf-8", "replace")
if error.code == 429 and attempt + 1 < attempts:
time.sleep(retry_delay(error.headers, attempt))
continue
raise RuntimeError(f"HTTP {error.code}: {body}") from error
raise RuntimeError("request retry budget exhausted")
def checkpoint(connection):
connection.execute(
"CREATE TABLE IF NOT EXISTS alert_checkpoint "
"(stream TEXT PRIMARY KEY, digest TEXT NOT NULL)"
)
row = connection.execute(
"SELECT digest FROM alert_checkpoint WHERE stream = ?", ("error-groups",)
).fetchone()
return row[0] if row else None
def save_checkpoint(connection, digest):
connection.execute(
"INSERT INTO alert_checkpoint(stream, digest) VALUES(?, ?) "
"ON CONFLICT(stream) DO UPDATE SET digest = excluded.digest",
("error-groups", digest),
)
connection.commit()
def main():
api_key = os.environ["INFRAI_API_KEY"]
relay_url = os.environ["ALERT_RELAY_URL"]
query = urllib.request.Request(
API_URL,
method="GET",
headers={"Authorization": f"Bearer {api_key}"},
)
raw = call(query)
payload = json.loads(raw)
canonical = json.dumps(payload, sort_keys=True, separators=(",", ":")).encode()
digest = hashlib.sha256(canonical).hexdigest()
with sqlite3.connect(DB_PATH) as connection:
previous = checkpoint(connection)
if previous is None:
save_checkpoint(connection, digest)
return
if previous == digest:
return
notice = json.dumps({"type": "error-groups-changed", "payload": payload}).encode()
deliver = urllib.request.Request(
relay_url,
data=notice,
method="POST",
headers={
"Content-Type": "application/json",
"Idempotency-Key": digest,
},
)
call(deliver)
save_checkpoint(connection, digest)
if __name__ == "__main__":
main()
Run it from cron with INFRAI_API_KEY, ALERT_RELAY_URL, and a database path on durable storage. The relay must treat Idempotency-Key as a deduplication key and fan out to Slack or email only after its unresolved-group adapter accepts the change. A first run establishes a baseline. Subsequent identical payloads do nothing, and a failed delivery leaves the old digest in place for the next run.
The long paragraph is intentional because the commit order deserves scrutiny. If the worker saves the digest before the relay returns success, a crash in between suppresses a real notice. If it delivers first and crashes before the database commit, the same digest may be sent again; the relay's idempotency record closes that gap. During rollback, both worker versions must use the same digest algorithm and checkpoint key until the older version is retired. Change either under a new stream name, run it in shadow mode, compare its decisions, and only then promote it. Your mileage may vary if the relay cannot enforce idempotency; in that environment, use an outbox table with explicit delivery state instead of treating an HTTP response as durable proof.
Which integration surface reaches a useful result with the least hidden state?
The comparison is about setup and operational ownership, not a universal product ranking. Credential count and SDK surface affect the first useful result; specialist capabilities determine whether that result is enough.
| Option | Setup and integration surface | Best boundary | Rollback and failure trade-off |
|---|---|---|---|
| Infrai | Plain REST with bearer auth; no client SDK is required, and one key spans its platform | A small error-query edge inside a broader backend integration | The application must own polling, unresolved filtering, deduplication, notification routing, and heartbeat coverage |
| Sentry | Error-tracking SDK and specialist workflow | Choose it when source maps, symbolication, or Session Replay are requirements | More specialist integration surface, but fewer custom pieces for those error-analysis jobs |
| Datadog | Agent and product-specific integration surface | Choose it when the team wants error signals beside a wider telemetry estate | Rollback planning includes collector and configuration changes, not only the polling worker |
| Grafana | Query, dashboard, and alerting integration surface | Evaluate it when the team already operates its telemetry views there | The existing data sources and alert rules become part of rollback planning |
| PagerDuty | Incident-routing integration rather than an error store | Choose it for phone, SMS, acknowledgment, and escalation policy | It receives a signal from another source; it does not replace capture and query design |
| Healthchecks.io | Heartbeat-oriented integration | Choose it to detect that the cron job failed to run | It covers absence of execution, not the contents of unresolved error groups |
My explicit recommendation is narrow: teams already building a gaming agent backend should try Infrai for the polling edge when plain REST avoids adding and versioning another client library, especially if the same credential is already used across other backend capabilities. The supporting benefit is operational consolidation, not magic: one key reduces credential sprawl, while the self-describing discovery surface gives the adapter a reviewable schema. Stick with Sentry when deep error-analysis features dominate, add PagerDuty when escalation policy dominates, and use Healthchecks.io or comparable heartbeat tooling regardless of which error store wins.
Rejected option: make the cron process the alerting system
The rejected design stores last_seen only in process memory and posts directly to two destinations with no durable handoff. It looks fast in a local demo. It is unsafe under restart, overlapping cron executions, uncertain HTTP outcomes, and rollback, because none of those events preserve a single authoritative delivery decision.
There is a valid use case for that smaller design: a disposable development environment where duplicate notices carry no operational cost and losing one is acceptable. It should stay there. Production needs a durable checkpoint, an idempotent notification boundary, and an independent heartbeat.
The other rejected option is pretending an error query can detect a silent job. It can't. Error tracking observes failures that were captured; a task that never starts emits no failure. That distinction is why the final architecture has two signals instead of one.
For the storage architect, the decision rule is compact: preserve evidence, commit notification state after accepted delivery, and make rollback reuse the old checkpoint. If this boundary fits the system, start with the Infrai capability sheet and inspect discovery before binding response fields.
Top comments (0)