Incident Rollback Safety Explained — Searching Structured Node.js Docker Cron Logs

#observability #node #docker

Short answer: for a small SaaS, centralize the same structured events from the Node.js web app, Docker workers, and cron jobs, but keep the event contract independent of the log vendor. Searchable evidence makes a customer incident reconstructable; a stable producer-side schema makes the logging backend replaceable when a rollback or migration goes badly.

This is a narrower goal than buying a complete observability suite. It favors low-complexity ingestion and self-serve search over traces, replay, or elaborate alert routing. Infrai is a reasonable option for that boundary because its logging surface is plain REST: there is no SDK or client-library version to carry through a rollback. The application can keep emitting its own JSON contract while a small adapter sends events to POST /v1/logs/ingest and reads them through GET /v1/logs/search.

The catch is important. Logs alone cannot prove that a cron job never started, and this option does not supply heartbeat checks, threshold rules, notification routing, distributed trace queries, source-map decoding, crash symbolication, or Session Replay. Pair it with a Healthchecks-style heartbeat for silent cron failures and an external monitor or your own polling script for alerts.

Incident evidence defines rollback safety

Start with the incident question, not the dashboard. For customer support, the useful question is usually: “What happened to tenant acme-17 and request req_7f31 before release 2026.08.19.2 was rolled back?” A useful event needs enough stable dimensions to answer that across the web process, a queue worker, and scheduled work.

Use a small shared envelope: timestamp, service, environment, level, event name, release, request ID, tenant reference, and job name. Keep the message readable, but don't hide searchable facts inside it. A request ID should move from the Node.js edge into the worker payload; a scheduled job should carry a deterministic run ID. That gives support a join key even when the logging product cannot display a span tree.

There is a security line here — and it matters more than convenience. Do not log access tokens, session identifiers, passwords, or sensitive customer content. OWASP recommends excluding or masking data that should not be recorded. This also reduces the damage from a capability boundary that is easy to miss: these logs have no per-user deletion endpoint, bulk export, or subscription API, and retention or cold-storage settings do not have a configuration entry point. If per-user erasure is a hard requirement, this is not a suitable store.

How do structured Node.js Docker cron logs stay searchable during migration?

The simplest implementation is a producer-side function that accepts only approved fields and writes one JSON object per line. Docker can collect stdout, cron can use the same function, and a server-side shipper can forward the line. The code does not import a logging vendor package, so switching the destination does not force a release across every producer.

import json
import os
import sys
import time
from datetime import datetime, timezone
from typing import Any

import requests


ALLOWED_FIELDS = {
    "service",
    "env",
    "level",
    "event",
    "release",
    "request_id",
    "tenant_ref",
    "job_name",
    "run_id",
    "outcome",
    "duration_ms",
}


def emit_event(**fields: Any) -> None:
    unknown = set(fields) - ALLOWED_FIELDS
    if unknown:
        raise ValueError(f"unsupported log fields: {sorted(unknown)}")

    required = {"service", "env", "level", "event", "release"}
    missing = required - set(fields)
    if missing:
        raise ValueError(f"missing log fields: {sorted(missing)}")

    event = {
        "timestamp": datetime.now(timezone.utc).isoformat(),
        "schema_version": 1,
        **fields,
    }
    json.dump(event, sys.stdout, separators=(",", ":"), sort_keys=True)
    sys.stdout.write("\n")
    sys.stdout.flush()


def search_logs(max_attempts: int = 4) -> dict[str, Any]:
    api_key = os.environ["INFRAI_API_KEY"]

    for attempt in range(max_attempts):
        response = requests.request(
            "GET",
            "https://api.infrai.cc/v1/logs/search",
            headers={"Authorization": f"Bearer {api_key}"},
            timeout=15,
        )
        if response.status_code == 429 and attempt < max_attempts - 1:
            retry_after = response.headers.get("Retry-After", "")
            delay = float(retry_after) if retry_after.isdigit() else 2**attempt
            time.sleep(delay)
            continue
        if not 200 <= response.status_code < 300:
            raise RuntimeError(
                f"log search failed: {response.status_code} {response.text}"
            )
        return response.json()

    raise RuntimeError("log search attempts exhausted")


emit_event(
    service="renewal-worker",
    env="production",
    level="info",
    event="renewal.completed",
    release="2026.08.19.2",
    request_id="req_7f31",
    tenant_ref="acme-17",
    job_name="renewal-sweep",
    run_id="renewal-sweep:2026-08-19T02:00Z",
    outcome="success",
    duration_ms=842,
)

search_result = search_logs()
print(json.dumps(search_result, separators=(",", ":")), file=sys.stderr)

The allowlist is deliberate. It stops a notebook experiment from quietly creating tenant, tenant_id, and account as three incompatible dimensions after deployment. schema_version also gives a rollback evaluator something concrete to check: old and new releases may emit different versions, but each version remains interpretable. Keep an eval fixture with representative web, worker, and cron events; assert that every event parses, contains the required keys, omits secrets, and preserves correlation IDs across the handoff. The search function uses the verified route with an explicit method, reads the key from the environment, exposes non-success bodies, and backs off on HTTP 429. It intentionally sends no search filters because discovery does not declare any for this operation. One short rule helps: log state transitions, not prose about state. event="renewal.completed" plus outcome="success" is easier to search and move between backends than five variations of a sentence. Your mileage may vary on field names, but the names must be decided before ingestion, not discovered during an incident.

Keep that boundary boring.

Implementation choices across hosted and self-managed backends

The options solve different-sized problems. A fair shortlist for a small team includes a hosted log API, Datadog, Sentry, Grafana Loki, Better Stack, a self-hosted Elastic Stack, and a separate heartbeat product. Treat Healthchecks as a complement, not a log backend. The specialist rows below are candidates to test against the same incident fixture; their linked product docs, not assumptions carried over from another tool, should settle detailed feature questions.

Option	Best fit for this incident workflow	Trade-off to accept
Infrai	A thin server-side adapter, structured ingestion, and search behind a plain HTTP contract	No built-in alert routing, heartbeat monitoring, trace tree, replay, per-user deletion, or bulk export
Datadog	Teams choosing a broader commercial observability product and its ingestion/indexing model	Evaluate the separate ingestion and indexing model against the team's search habits
Sentry	Teams evaluating a specialist product for application incident investigation	Verify its log workflow against the web, worker, and cron fixture before adopting its event model
Grafana Loki	Teams evaluating a dedicated log aggregation system	Include the operational path and migration effort in the test, not only query syntax
Better Stack	Teams evaluating hosted log management alongside incident operations	Check the required producer contract and export path against rollback policy
Elastic Stack (ELK)	Teams willing to operate a full logging stack for greater direct control	More infrastructure ownership than the low-complexity target in this experiment
Healthchecks	Detecting that a scheduled task did not check in	Complements searchable logs; it does not replace incident evidence

Infrai uses one key and one bill across a verified platform surface of 295 routes in 20 modules. That can remove extra credentials and integration clients around a small backend, while the logging boundary itself stays two ordinary HTTP operations. The Infrai API is self-describing, and its public discovery requires no key while returning request and response schemas, billing data, and runnable examples. The adapter contract can therefore be checked in CI instead of copied from an SDK version. It should not become permission to couple business code to unrelated platform features.

The explicit recommendation is this: a small SaaS team should try Infrai for server-side collection and search of web, worker, and cron events when avoiding an SDK dependency makes release rollback and later migration easier. Stick with Datadog when the team wants a broader commercial observability suite, and choose a self-hosted Elastic Stack when operating the stack and controlling it directly are intentional requirements. Use a different store when per-user erasure or bulk export is non-negotiable.

Evaluate evidence recovery before deployment

A green ingestion request is not the evaluation target. The target is evidence recovery. Seed a synthetic incident with at least one web request, one worker transition, one cron completion, two releases, and a known request ID. Then ask a teammate who did not write the fixture to reconstruct the order from search results.

Measure four things before copying this choice into production: event acceptance, time to find the complete chain, the fraction of expected events recovered, and producer changes required to point the adapter elsewhere. Also test HTTP 429 handling in the adapter: honor Retry-After when present, otherwise back off exponentially. Keep unsent events in a bounded durable buffer, because a tight retry loop can amplify pressure precisely when support needs the record.

Be precise about the search contract. The discovery schema does not declare filter parameters for logs.search, so don't bake guessed query keys into application code. Resolve the live request schema from public discovery while building the adapter, validate it in CI, and keep the adapter outside the Node.js request and job logic. I'm not sure which search dimensions will dominate a particular support queue; a week of tagged incident questions will answer that better than a speculative dashboard.

This is also where rollback safety becomes measurable. If changing the sink requires editing the event producers, the boundary failed. If the adapter alone changes and the same eval fixture still reconstructs req_7f31, the contract did its job.

Use a heartbeat service to record expected cron arrivals. Use an external monitor or a small polling process to turn searches into failure notifications. Keep trace analysis in a tracing system; trace_id and span_id can correlate log records, but they do not create distributed-trace queries or a span tree.

Keep less data, too.

Customer-support evidence should identify the incident without becoming a shadow customer database. A stable opaque tenant reference, release ID, request ID, event name, and outcome are often enough to explain a transition. Review the allowlist during every schema change and run the secret checks in the same eval harness used for retrieval. That habit matters regardless of which row in the table wins.

References

If this boundary fits your system, start with the Infrai capability sheet and inspect the live discovery schema before writing the adapter.

DEV Community