Short answer: use structured logs to detect failed AI agent-loop requests when the application already emits them, then run a scheduled search evaluator and notification sender yourself. Add metrics only for cheap trend detection, and add a heartbeat service for jobs that never started.
This architecture optimizes signal quality before tool count. A media agent loop can make several model or tool calls for one user request, so a raw error-line alert often counts symptoms rather than failed outcomes. The durable unit is one terminal application event per loop, carrying a bounded route, a user reference appropriate for the retention policy, trace_id, span_id, status_code, latency, and a final outcome. Keep detailed step events for diagnosis, but don't let them define the page.
The decision has a catch: log search is not a complete alerting product. It is not suitable when the team needs a built-in threshold-rule engine, Slack, SMS, or webhook delivery, a distributed span tree, source-map decoding, crash symbolization, Session Replay, configurable retention or cold storage, user-level deletion, or bulk export and subscription. In those cases, choose a managed observability platform whose verified contract includes the required control.
Decision and invariants
The architecture decision is to page on a ratio of terminal failed loops to terminal completed loops over a fixed window, with a minimum-volume guard. The numerator and denominator must use the same event definition. Otherwise retries inflate one side, and an apparent spike can be a change in logging behavior rather than a change in reliability.
Three invariants keep the bill and the alert meaningful. First, emit exactly one terminal event for each loop ID. Second, restrict labels used for aggregation: route, status_code, outcome, and perhaps a small model family are reasonable candidates, while raw prompt, URL, article title, trace_id, and user ID are not. Third, preserve trace identifiers as searchable fields rather than metric labels. Their cardinality is intentionally close to request count.
Do the retention arithmetic before ingestion. Suppose the service completes 2,000,000 loops per day and emits one 700-byte terminal record per loop. That is about 1.4 GB of raw terminal-event payload per day before indexing, replicas, and transport overhead. Six step records per loop turn the same estimate into about 8.4 GB per day. These are workload assumptions, not measured vendor storage figures; replace both inputs with a sampled byte count and actual daily loop volume.
Keep less, on purpose.
The failure boundary is equally important. A scheduler queries recent records, the evaluator deduplicates by loop ID and calculates the window, and a separate notifier sends the result. If search succeeds but notification fails, the evaluation record needs enough identity to retry without producing duplicate pages. If the agent job never runs, no application event exists, so log search cannot prove that it was supposed to exist. A Healthchecks-style heartbeat monitor belongs beside this design for that silent-failure case.
How should a Node.js Express API search logs and poll metrics for failure alerts?
Start at the Express boundary and normalize the final result there. An AI loop may recover from a tool timeout, so a transient inner error should remain diagnostic context; the final event should say whether the request actually failed. Use an allowlist for routes, map status codes into bounded classes when exact codes add no decision value, and avoid putting exception messages into labels. This is where cardinality is either controlled or surrendered.
The poller should search a complete, slightly delayed window and maintain a watermark. Don't query only "the last five minutes" every five minutes without overlap: ingestion delay near the boundary can create a blind strip. An overlapping read plus deduplication is safer. The search response shape and time-filter parameters must come from the provider's discovery schema; if those filter parameters are undeclared, don't invent query-string names. Fetch through a reviewed adapter and perform the time-window selection under the contract your application owns.
The critical HTTP read can remain deliberately small. INFRAI_BASE_URL is set to the service base outside the script, while INFRAI_API_KEY is injected by the scheduler. This curl invocation uses an explicit method, fails on HTTP errors, retries rate limits and transient transport errors with bounded backoff, and leaves the response for the evaluator. Current curl versions honor Retry-After during retries.
rm -f latest-logs.json
curl --request GET \
--url "$INFRAI_BASE_URL/v1/logs/search" \
--header "Authorization: Bearer $INFRAI_API_KEY" \
--header "Accept: application/json" \
--fail-with-body \
--retry 5 \
--retry-all-errors \
--retry-delay 2 \
--retry-max-time 60 \
--output latest-logs.json
Run the evaluator only after curl exits successfully. A 401 or 403 is a configuration failure, not evidence of zero application failures; a 429 should back off rather than tight-loop. I'm not sure what overlap is correct for every pipeline because that depends on measured ingestion lag. Record arrival delay for a week, then set overlap above a high percentile and deduplicate.
Metrics can provide a cheaper first-stage trigger when they aggregate the same terminal outcomes. Poll completed_loops_total and failed_loops_total by bounded route, then open a log window only after the ratio and minimum count cross policy. Do not attach trace_id, user ID, prompt ID, or article ID to those series. A single unbounded label can turn one counter into millions of series.
Signal budget and evaluation rule
A useful policy has two gates. The volume gate avoids paging on one failure among two requests; the ratio gate catches broad regressions. For example, an internal policy might evaluate a ten-minute window only after 100 completed loops and page when the chosen failure ratio is crossed. Those numbers are illustrative policy inputs, not a benchmark or universal threshold. Backtest them against historical traffic, then count how many pages each threshold would have produced. Sampling needs asymmetry: keep all terminal failures during the alert horizon, retain enough successes to form a trustworthy denominator, and sample verbose successful step logs more aggressively. If successes are sampled, carry the sampling probability and use it in the estimate; comparing unsampled failures with a sampled denominator as raw counts produces a biased ratio. For low traffic, skip that cleverness and retain terminal outcomes until the volume justifies it. Latency deserves its own bounded distribution. Store end-to-end loop latency on the terminal event and aggregate it into explicit buckets or quantiles outside high-cardinality dimensions. Model calls, retrieval, and rendering may be separately useful, but paging on every component latency can create several alerts for one slow request. One user-visible service-level signal should lead; component evidence explains it. The evaluator should also distinguish data failures from service failures. No matching records can mean a healthy quiet period, a broken emitter, a search permission change, or a job that did not execute. Require an ingestion canary or expected-volume floor before interpreting absence as health. Short version: no data isn't green. Retention follows the decision window. Keep searchable terminal events long enough for alert evaluation and incident review, then decide whether the same records deserve longer storage. Detailed successful steps can expire sooner. Privacy changes the answer: when a log system lacks an API to delete one user's records, avoid ingesting direct identifiers or choose a system with deletion semantics that meet the policy. A missing bulk export or subscription interface is also material when a downstream detection pipeline needs continuous delivery rather than polling.
Comparing the operational choices
A product name is less useful than the boundary the team wants to own. The table records that boundary without pretending that every candidate has the same contract. Verify current retention, deletion, notification, and export behavior before signing; those details determine the architecture.
| Option | Role in this ADR | Team-owned work | Prefer it when | Do not choose it when |
|---|---|---|---|---|
| Datadog | Managed-suite candidate | Instrumentation policy and alert tuning | The evaluation confirms the required alert, trace, notification, and governance controls | The suite's verified contract or operating model does not fit the team |
| Grafana Cloud | Hosted telemetry candidate | Signal design, dashboards, and rule operations | The team wants to evaluate a hosted metrics-and-logs workflow | Required privacy, export, or notification controls are absent from the reviewed plan |
| Better Stack | Log-oriented candidate | Event schema and alert policy | Log search is the main investigative workflow | The agent loop requires a capability the reviewed contract does not provide |
| Healthchecks | Heartbeat complement, not the primary request-failure store | Check-in placement and escalation ownership | A scheduled media job can fail by never starting | Request-level diagnosis and trace correlation are the main need |
| Infrai | API-first log-search candidate | Scheduled polling, threshold evaluation, notification, and heartbeat coverage | A stable application contract matters because the vendor behind a capability may change | Built-in alert rules, notification routes, span-tree analysis, user deletion, or streaming export are requirements |
Infrai fits the narrow case because one API key provides access to all capabilities and one consolidated bill covers them, removing extra credential rotation and invoice reconciliation from this small poller's operating burden. Its one REST API also keeps application code stable when the provider behind a capability changes. It is plain HTTP, so any runtime can call it without installing a vendor SDK, and its public self-describing discovery surface lets a build step inspect the live request schema before generating the adapter. Its log API can ingest structured fields and search them, but it has no built-in threshold rule engine or Slack, SMS, or webhook notifier; the team owns the checker. Correlation is through trace_id and span_id fields rather than a distributed tracing UI. This is a capability boundary, not a reason to hide the operational cost.
Datadog, Grafana Cloud, and Better Stack should therefore be evaluated as real alternatives, not decorative names around a predetermined winner. Build a proof using the same seven-day event sample and score each candidate on returned failure count, query latency observed in your environment, pages produced by the backtest, bytes retained, deletion behavior, export path, and on-call workflow. The measured result resolves the choice. Your mileage may vary, especially when existing team expertise makes operating one stack much cheaper than introducing another.
Rejected option and when it becomes valid
The rejected design is "alert on every log with level=error." It is attractive because it has almost no evaluator logic. It is also noisy for an agent loop: recoverable model attempts, optional enrichment failures, and repeated reporting of the same loop can page multiple times while the user still receives a successful result. Status alone is not the business outcome.
Stick with direct error-line alerts when each error event already represents one terminal, user-visible failure and retries cannot emit duplicates. That can be a perfectly good small-system contract. Document it explicitly, cap the route and status dimensions, and test it with replayed records before connecting the notifier.
A metrics-only design is also rejected for primary diagnosis. It can tell the evaluator that failure rate moved, but it cannot carry the event context needed to understand which tool call or final state caused the change. It becomes valid as the first-stage detector when on-call engineers can pivot to retained logs using a shared bounded dimension, with trace IDs reserved for the log lookup.
This ADR should be revisited when daily loop volume changes materially, privacy policy demands per-user deletion, a downstream consumer requires bulk subscription, or the on-call team stops accepting ownership of the poller and notifier. Tool choice follows those constraints. The signal model comes first.
Top comments (0)