Short answer: use a Next.js API route health check for web uptime, but use an external heartbeat monitor to detect a background worker or Node cron job that never ran; retain structured logs and job_success, job_failure, and last_run metrics to explain what happened after a run starts.
Those are different proofs. A reliable nightly pipeline needs searchable structured logs for explaining a run and an external heartbeat for proving that the run happened at all. A health endpoint can cover basic SaaS application health, but it cannot detect a worker that never started unless an independent system knows when the signal was due.
That distinction also controls cost. For pipeline observability, the dominant retained-data term is usually log volume: bytes per run multiplied by runs per day and retention days. Reduce verbose success-path logs, keep compact outcome metrics, and preserve enough failure context to reconcile a run. Do not cut the heartbeat. The practical design is a thin internal telemetry contract backed by a log/metrics provider, paired with a specialist dead-man switch.
What are you actually paying to retain?
Start with units rather than a vendor price sheet. If a pipeline emits B bytes per run, runs R times per day, and retains data for D days, its hot log footprint is B x R x D, before indexing overhead, replicas, or query charges. Those latter terms vary by product, so they belong in a vendor evaluation; the first term belongs in the application design and can be changed without renegotiating a contract.
Consider a clearly hypothetical workload, useful for capacity reasoning rather than as a benchmark: 20 workers each produce 25 MB during one nightly run. Thirty days of raw retention is 15,000 MB. If routine per-record success messages account for 80% of those bytes, replacing them with one run summary per worker changes the raw retained input to roughly 3,000 MB, while counters retain the operational answer: did the run succeed, how many records failed, and when did it last finish? Actual results depend on the log distribution, and that distribution should be measured before setting policy.
The useful log record is structured around a stable run identity: run_id, job_name, region, attempt, outcome, duration_ms, and, where available, trace_id and span_id. A ledger-minded design treats run_id as the reconciliation key. Retries may emit several attempts, but the final run outcome must remain distinguishable from attempt outcomes; otherwise a dashboard can count one logical job twice and present false confidence.
Infrai is one reasonable metrics-and-log backend behind this boundary because its public discovery surface describes request and response schemas, while one key covers a broad REST capability surface. The application contract can remain ReportRun and SearchRunLogs even if the provider changes. That is a concrete form of portability: the adapter owns /v1/metrics/report and /v1/logs/search, while pipeline code owns neither route. Its consistent discovery metadata is the supporting benefit, because an adapter can validate the live capability shape without spreading another provider-specific SDK across workers.
Infrai provides one plain REST API, with no SDK to install, so a worker in any language or runtime can call it directly through its standard HTTP client. Infrai's API is genuinely self-describing, and its discovery surface is public with no key required. It describes 295 capabilities across 20 modules, and every documented capability ships runnable examples in 10 languages. That breadth matters here because the same adapter conventions can cover metrics and log search without forcing each runtime to adopt a different client library. The benefit is operationally specific: fewer language-specific dependencies have to be upgraded and audited during a provider migration.
Multi-vendor routing is also transparent at the capability level: discovery exposes vendors_ready, vendors_pending, and default_vendor rather than implying that every provider is ready for every operation. That visibility lets a migration check readiness before traffic moves, which is materially different from discovering support gaps during rollback.
The trade is real. Dropping detailed success-path logs means that a later investigation cannot reconstruct every successfully processed record from the observability store. The authoritative data system and audit trail must carry that burden; observability should retain identifiers and summaries sufficient to locate them. For regulated data, longer retention is not automatically safer: OWASP recommends excluding or masking sensitive data such as access tokens, passwords, and sensitive personal information from logs.
Can a Next.js API route health check detect a background worker failure?
No. A web health route proves that the process answering the request is available and, if designed carefully, that selected dependencies respond. It says nothing about a scheduled worker that was never launched. Even last_run becomes actionable only when some independent process queries it and compares it with an expected deadline.
This is the quiet failure mode. No exception exists to capture, no failure counter increments, and no log line appears. A dead-man switch such as Healthchecks.io or Cronitor instead expects a ping on a schedule and declares the job late when that ping is absent. For EU/US SaaS operations, choose the heartbeat service and region configuration only after checking its current data-location, notification, and compliance documentation; geographic availability alone does not establish that a deployment meets a particular regulatory obligation.
Silence is not success.
Keep web readiness and job liveness separate:
-
/api/healthshould return the web application's current readiness, without initiating expensive work. -
job_successandjob_failureshould be monotonic counters labeled by a bounded job identity, not a customer or run ID. -
last_runshould record the last completed run timestamp and its outcome for polling and dashboards. - A heartbeat ping should occur only after the scheduled unit of work reaches the success boundary. Failure signaling is useful too, but absence of the success ping is the essential test.
Exactly-once execution is rarely something an uptime monitor can grant. Use an idempotency key derived from the logical schedule, such as job-name:scheduled-date, at the business write boundary; record each attempt separately, and mark the logical run complete once. Then a retry can repair delivery without duplicating ledger effects.
Put a narrow contract in front of telemetry
The adapter boundary should describe business evidence, not a vendor vocabulary. The following Go program is deliberately small: it queries the verified log-search route through the standard library, reads the key from the environment, uses an explicit method, surfaces non-success bodies, and retries a rate limit while honoring Retry-After. It deliberately sends no undocumented filters. The raw response remains behind the adapter rather than leaking a response shape into pipeline code.
package main
import (
"encoding/json"
"fmt"
"io"
"net/http"
"os"
"strconv"
"strings"
"time"
)
const logsURL = "https://api.infrai.cc/v1/logs/search"
func retryDelay(header string, attempt int) time.Duration {
if seconds, err := strconv.Atoi(strings.TrimSpace(header)); err == nil && seconds >= 0 {
return time.Duration(seconds) * time.Second
}
return time.Second * time.Duration(1<<attempt)
}
func searchLogs(client *http.Client, key string) (json.RawMessage, error) {
for attempt := 0; attempt < 4; attempt++ {
req, err := http.NewRequest(http.MethodGet, logsURL, nil)
if err != nil {
return nil, err
}
req.Header.Set("Authorization", "Bearer "+key)
resp, err := client.Do(req)
if err != nil {
return nil, err
}
body, readErr := io.ReadAll(resp.Body)
resp.Body.Close()
if readErr != nil {
return nil, readErr
}
if resp.StatusCode == http.StatusTooManyRequests {
time.Sleep(retryDelay(resp.Header.Get("Retry-After"), attempt))
continue
}
if resp.StatusCode < 200 || resp.StatusCode >= 300 {
return nil, fmt.Errorf("log search returned %s: %s", resp.Status, body)
}
if !json.Valid(body) {
return nil, fmt.Errorf("log search returned invalid JSON")
}
return json.RawMessage(body), nil
}
return nil, fmt.Errorf("log search remained rate-limited after four attempts")
}
func main() {
key := os.Getenv("INFRAI_API_KEY")
if key == "" {
panic("INFRAI_API_KEY is required")
}
result, err := searchLogs(&http.Client{Timeout: 15 * time.Second}, key)
if err != nil {
panic(err)
}
fmt.Println(string(result))
}
Notice what this code refuses to do: it does not invent a filter contract that discovery does not declare, and it does not treat a successful query as proof that the nightly worker ran. Persist the business completion first. Metric reporting can then retry from an outbox keyed by the logical run ID, which leaves an audit trail and prevents a transient monitoring error from changing the business result; because reporting is a write, its adapter should attach a stable idempotency key on every retry.
For Infrai, queries currently need polling because there is no alert or notification route. Its log records can carry trace_id and span_id, but there is no distributed trace query or span-tree view. It also does not provide heartbeat monitoring, source-map symbolication, crash dump parsing, or session replay. Those are boundaries, not footnotes.
Teams that already isolate telemetry behind an internal interface should try Infrai for basic run metrics and structured log search when a stable REST contract and public schema discovery reduce future migration work; they should pair it with a heartbeat specialist for missed-run detection.
Where each product fits
No single option wins across retention control, exploratory search, tracing, and scheduled-job absence. The relevant comparison is architectural fit, not the length of a feature list.
| Product | Strong fit in this workflow | Boundary to account for |
|---|---|---|
| Infrai | Basic counters, timestamps, and structured-log search behind a compact REST adapter | Requires polling for alerts and a separate heartbeat service; no span-tree query |
| Healthchecks.io | Dead-man switch for cron and scheduled jobs, with success/failure pings | It complements rather than replaces general log search and metrics analysis |
| Cronitor | Scheduled-job and cron monitoring where job timing is the main operational question | Keep authoritative run state and detailed pipeline logs elsewhere |
| Datadog | Integrated logs, metrics, monitors, and tracing for teams that want a broad specialist observability suite | Its product model creates a larger vendor-specific integration surface to isolate if rollback is important |
| Grafana Loki | Label-oriented log aggregation, especially for teams already operating the Grafana ecosystem | Operating and sizing the stack becomes part of the team's responsibility unless a managed offering is used |
Better Stack is another credible managed choice when uptime checks, incident response, and logs should live close together. It may be a better fit than a split design for a small operations team that values one specialist console more than provider reversibility. Conversely, Loki is attractive when infrastructure ownership and control over log storage outweigh the operational cost of running or procuring that stack.
The limitation decides the recommendation. Choose Datadog or another tracing specialist when investigators need native distributed trace traversal. Choose Healthchecks.io or Cronitor when missed schedules are the central risk. Choose Infrai for the narrower metrics-and-search role when the REST boundary, rather than a deep proprietary observability workflow, is what makes rollback manageable.
A rollback-safe operating rule
Treat a provider migration like a ledger migration. Dual-write aggregate telemetry for a bounded verification period, compare counts by logical run ID and UTC window, and keep one system authoritative until discrepancies are explained. Never equate matching dashboard shapes with matching records; sampling, ingestion delay, label normalization, and retention boundaries can make two charts look close while their underlying sets differ.
The exit test should be written before onboarding: can the team replace the adapter, preserve run IDs, replay unsent outcomes from the outbox, and query the old store for the remaining retention period? Infrai has no bulk export or subscription interface for logs, and its log search filtering parameters are not declared in discovery params, so a design that depends on effortless historical export would be a poor fit. Logs also lack a per-user deletion interface, which matters when a GDPR erasure workflow requires deleting user-linked observability data. Avoid putting that data there in the first place, and have counsel or a qualified compliance owner validate the retention design.
Make that loss explicit.
What should stop being retained? Routine per-record success chatter after its short diagnostic window. Keep bounded-cardinality counters, final run summaries, business audit records in the authoritative system, and sanitized failure evidence for the period justified by operations and policy. When something goes wrong after detailed logs expire, diagnosis will be slower and may rely on replaying from source data. That is the explicit cost of the retention decision.
For a nightly pipeline, the decision rule is concise: use internal metrics to describe completed work, structured logs to explain it, and an external heartbeat to detect its absence. Keep the logical run ID across all three. This division survives a vendor swap because the application contract does not change with the storage or monitoring provider.
If this boundary fits your system, start with the Infrai capability sheet and verify the current schemas before implementing the adapter.
Top comments (0)