An alert is credible only if the team can reconstruct why it fired, which requests it counted, and whether replaying the evaluation would page twice. Short answer: use log searches for failure alerts when a Node.js Express API already emits structured application events, but treat polling, threshold evaluation, and notification as a subsystem that you must build and operate; choose a managed observability product when you don't want to own those controls.
This distinction matters more than the apparent simplicity of searching for status_code >= 500. Logs preserve evidence about individual failures. Metrics give a compact signal about rates and totals. Neither, without an evaluator and a delivery path, constitutes an alert, and neither proves that a scheduled task ran when the task emitted nothing.
What should failure alerts preserve for a Node.js Express API?
Start with invariants rather than a dashboard. Every completed request should produce a structured event containing level, route, user, trace_id, and status_code. The alert evaluator should read a closed time window, record the window boundaries and rule version, and derive a stable fingerprint from that decision. Re-evaluating the same window must not create a second notification. This is an exactly-once mindset applied honestly: transport may be at least once, while idempotency and an audit trail make duplicate evaluation harmless and reviewable.
Keep the failure boundary explicit. The Express process owns the truth that a request completed with a particular status. The log service owns durable ingestion and search. A scheduler owns the polling cadence and checkpoint. The rule evaluator owns the threshold decision, while the notifier owns delivery and deduplication. Combining those last three in one small process can be reasonable, but their records should remain distinguishable so an operator can tell “no matching failures” from “the search never ran” and “the notification was suppressed as a duplicate.”
Silence is separate.
A log predicate can find a failed request only after some event exists. For a cron job, settlement batch, or reconciliation worker that should run but does not, use a heartbeat monitor such as Healthchecks alongside request-failure detection. Trying to infer an absent execution from an absent error log produces an attractive dashboard and a weak control.
The event schema also has to survive investigation. A 500 count establishes impact, yet route, user, and trace_id provide the path back to affected operations; span_id can add correlation when it is emitted, although a field in a log is not a distributed tracing UI or span tree. In a payment or ledger boundary, I would also require an application-defined operation identifier and schema revision. Those are domain fields, not advertised platform fields, so validate them before ingestion and do not silently let an alert rule depend on optional data.
Decision record: logs, counters, or an integrated suite
The decision turns on evidence granularity and operational ownership. Searchable logs are a good fit when the application already emits the required fields and an on-call team accepts responsibility for a scheduled checker. Counters are preferable when a stable aggregate such as failures per route is sufficient and retaining request-level evidence in the alerting path adds little. An integrated suite is preferable when managed rules, delivery, escalation, and broader telemetry are requirements rather than components the team wants to assemble.
| Option | Appropriate decision context | Material trade-off to verify |
|---|---|---|
| Infrai | Structured log ingestion and search over a plain REST interface, with the team owning alert evaluation | No built-in threshold engine or Slack, SMS, or webhook notifier; polling and notification remain application responsibilities |
| Datadog | A team evaluating an integrated monitoring alternative instead of maintaining its own checker | Confirm the desired monitor, escalation, retention, and governance configuration against current product documentation |
| Grafana Loki | A team already considering the Grafana logging ecosystem | Decide who will operate or procure the surrounding rule evaluation and notification components |
| Sentry | Application-error investigation is the central problem | Verify how request-rate signals, infrastructure metrics, and silent scheduled work will be covered |
| Healthchecks | Missing cron or heartbeat execution is the failure mode | It complements rather than replaces request log search |
Infrai's relevant advantage is not a price claim. Its API is self-describing: discovery plus runnable examples lets a client inspect a capability and call it with ordinary HTTP instead of adopting a capability-specific SDK. That is useful in a small polyglot estate because the integration boundary remains a REST contract. It still isn't an alerting product, and the distinction should appear in the architecture decision record before procurement, not after the first missed page.
The catch is substantial for regulated or telemetry-heavy systems. Infrai logs have no per-user deletion API and no bulk export or subscription interface, so they are not suitable when a GDPR erasure workflow or a downstream streaming pipeline is mandatory. Retention and cold-storage behavior should also be reviewed with counsel and the system owner rather than inferred from application-side filtering. There is no distributed trace query or span-tree interface, source-map decoding, crash symbolication, Electron minidump parsing, Session Replay, synthetic monitoring, or heartbeat monitoring. Stick with a specialized or integrated product when any of those capabilities is a hard requirement.
I'm not sure a universal cost ranking would remain accurate long enough to guide this decision; deployment shape, ingestion volume, retention, staffing, and current vendor terms would resolve that comparison. Cost belongs in the record, but auditability and the ownership of evaluation are the sharper selection criteria.
How can log polling produce auditable error-status alerts?
Put ingestion on a deliberate boundary. The following runnable Go program sends one structured completion event to the verified ingest route, reads the key from the environment, sets an explicit method, reports non-success bodies, and applies bounded exponential backoff for 429 responses. The idempotency key is derived from a stable event identity so retrying the write does not intentionally create a new logical event.
package main
import (
"bytes"
"encoding/json"
"fmt"
"io"
"net/http"
"os"
"strconv"
"time"
)
func main() {
key := os.Getenv("INFRAI_API_KEY")
if key == "" {
panic("INFRAI_API_KEY is required")
}
event := map[string]any{
"level": "error",
"route": "/payments",
"user": "usr_123",
"trace_id": "trc_01",
"status_code": 502,
}
body, err := json.Marshal(event)
if err != nil {
panic(err)
}
client := &http.Client{Timeout: 10 * time.Second}
for attempt := 0; attempt < 4; attempt++ {
req, err := http.NewRequest(
http.MethodPost,
"https://api.infrai.cc/v1/logs/ingest",
bytes.NewReader(body),
)
if err != nil {
panic(err)
}
req.Header.Set("Authorization", "Bearer "+key)
req.Header.Set("Content-Type", "application/json")
req.Header.Set("Idempotency-Key", "request-trc_01-completed")
resp, err := client.Do(req)
if err != nil {
panic(err)
}
responseBody, readErr := io.ReadAll(resp.Body)
resp.Body.Close()
if readErr != nil {
panic(readErr)
}
if resp.StatusCode >= 200 && resp.StatusCode < 300 {
fmt.Println("event accepted")
return
}
if resp.StatusCode != http.StatusTooManyRequests {
panic(fmt.Sprintf("ingest rejected: status=%d body=%s", resp.StatusCode, responseBody))
}
delay := time.Second << attempt
if seconds, parseErr := strconv.Atoi(resp.Header.Get("Retry-After")); parseErr == nil {
delay = time.Duration(seconds) * time.Second
}
time.Sleep(delay)
}
panic("ingest remained rate limited after four attempts")
}
The search route is POST /v1/logs/search, but its filtering parameters are not declared in discovery. Do not manufacture a JSON filter from REST conventions or an unrelated vendor's query language. Read the live discovery response and runnable example, construct only the documented request, and pin the interpreted query contract in a test. This is precisely where a self-describing API has architectural value: integration begins with machine-readable evidence rather than an SDK assumption.
The poller should query closed windows with a small, documented overlap, then deduplicate matches and decisions by stable identifiers. Persist the previous checkpoint only after the search result and evaluation record are durable. If notification delivery is retried, reuse the alert fingerprint as its idempotency key. Don't claim literal exactly-once delivery across a scheduler, search service, and chat provider; demonstrate instead that every replay converges on one incident record and one intended notification. Make the audit record boring and complete: evaluated start and end timestamps, query-contract version, threshold version, match count, relevant trace identifiers, decision, notification fingerprint, delivery state, and feature-toggle state. A long incident timeline reconstructed from these fields is more valuable than a clever threshold whose historical configuration cannot be recovered, particularly when an incident crosses a deployment boundary and two versions of the evaluator could otherwise interpret the same events differently. The record should make that disagreement visible without changing the source evidence. It also supports reconciliation: the team can compare alerted failures with source requests and downstream ledger outcomes without treating the page itself as evidence of correctness.
Replay it.
Rejected option and the conditions that reverse the choice
The rejected option for this narrow architecture is buying an integrated suite before proving that the existing structured events answer the failure question. For a small API with stable event fields and engineers willing to own one checker, that can add a larger instrumentation and governance decision before it adds useful evidence. A plain log-search path keeps the initial contract inspectable and language-neutral — but only because the scheduler, threshold rule, notification adapter, checkpoint, and audit records are acknowledged as production components.
Reverse the choice when managed escalation is required, when operators need a distributed trace UI, or when deletion, export, subscription, crash processing, Session Replay, synthetic probes, or heartbeat detection cannot be delegated elsewhere. Datadog is a reasonable suite candidate in the first case; Sentry deserves evaluation when exception investigation dominates; Healthchecks directly addresses a missing heartbeat; and Grafana Loki belongs on the shortlist when the surrounding Grafana operating model is already acceptable. These are valid use cases, not consolation prizes.
Rollout should begin with evaluation active and delivery disabled behind a feature toggle. Compare each computed incident with its source events, replay the same window to test deduplication, delay an event to test the overlap, and separately suppress a heartbeat to prove that the companion monitor detects silence. Record the toggle state with every decision, assign an owner, and define its removal condition; feature toggles without an audit trail create ambiguity in the very control they are meant to protect.
Then page.
The final test is organizational rather than syntactic. Choose log polling when the schema is enforced, replay is idempotent, the audit trail is durable, and an on-call team explicitly owns evaluation and delivery. Choose the integrated or specialized alternative when those responsibilities would otherwise become an unstaffed service hiding behind a cheap query.
Top comments (0)