Adding Centralized Application Logs to a Small Node.js SaaS Marketplace Backend

Send structured application events from every service to one searchable store, but alert on the absence of a committed import result rather than on the absence of an error. That is the shortest safe path for a small marketplace backend: emit newline-delimited JSON to standard output, attach a stable import execution ID, centralize collection outside the application process, and preserve the old event contract during rollback.

Short answer: treat centralized logging as an evidence pipeline, not as remote printf. For a scheduled catalog import, the decisive evidence is a terminal import_result event whose execution ID can be reconciled with the scheduler's start record. Error search remains useful, yet it cannot prove that work completed; a worker can hang, exit between stages, or accept an empty upstream response without ever producing an error. The monitoring model should therefore cover both failures and traffic, two of the signals identified in the Google SRE monitoring guidance [1], while the deployment model should make reverting an emitter independent of reverting collection.

This ADR chooses that contract because rollback safety is the primary decision axis. It applies equally when the application tier includes Node.js, FastAPI, or Rails: framework-specific logger configuration ends at a common JSON envelope, and the collector owns transport, buffering, and destination credentials.

How should a small SaaS add centralized application logs?

The invariant is narrow: for each scheduled execution accepted by a worker, exactly one terminal result must be explainable. “Exactly once” here is an audit objective, not a claim that a distributed runtime magically provides exactly-once delivery. Duplicate events are permitted in transport, so the search and alert layer deduplicates by event_id; conflicting terminal outcomes for one execution_id are a correctness alarm. In a concrete two-stage check, an execution that has import_scheduled but lacks import_started points toward dispatch or worker availability, while an execution that has started but has neither terminal event points toward worker progress. A separate reconciliation pass compares those log facts with durable imported records, because the absence query alone cannot tell whether the business commit happened just before a process exit. This division produces more work than a single error search, but each alarm now names the boundary that an operator must inspect.

Silence is ambiguous.

Use four event kinds: import_scheduled, import_started, import_result, and import_failed. The result carries counts for records read, accepted, and rejected. Zero is data. An import that correctly reads an empty supplier feed must emit a successful result with zero counts, because silence would otherwise be indistinguishable from a stalled worker.

Several fields are invariants across languages and releases: UTC timestamp, schema version, event name, event ID, execution ID, service, deployment revision, and outcome. Begin with schema version 1. Marketplace identifiers may be present when operationally necessary, but credentials, raw supplier payloads, payment data, and customer secrets do not belong in the event. Retention and access must follow the organization's compliance policy; the application contract should classify fields so that an operator is not forced to infer sensitivity during an incident.

The failure boundaries are intentionally separate. The scheduler can fail before dispatch. A queue can accept work that no worker begins. A worker can begin without reaching a terminal state. The application can write a terminal event that the collector never delivers. Finally, the searchable store can receive the event while the alert query is wrong. One generic “job failed” alert collapses those boundaries and weakens rollback decisions.

Decision and alternatives

The chosen design is structured standard-output events, a process-external collector, and a central searchable store with an absence query keyed by schedule and execution ID. The store may be self-hosted or managed. The contract does not depend on that procurement choice.

Option	Application coupling	Evidence during destination failure	Rollback behavior	Decision
JSON to standard output, external collection	Low; applications own only the event schema	Collector buffering can be tested independently	Old and new application revisions can emit the same schema	Chosen
Application sends directly to a remote log API	High; retry, authentication, and backpressure enter request or worker code	Depends on each language client and its queue	Reverting application code may also revert transport behavior	Rejected for this system
Search local files on each host	Tied to host lifecycle and access	Evidence remains fragmented	A rollback can move work to a different host	Valid only for a single durable host with manual operations
Infer health from metrics alone	Low log volume	Aggregate state survives if metrics delivery works	Easy to keep stable across revisions	Complementary, but insufficient for record-level reconciliation

This is not a claim that one storage engine is universally best. For a small team without a dedicated operations role, the meaningful evaluation is whether an operator can search exact execution IDs, constrain access, define retention, export evidence, and test delayed or duplicate delivery without changing application code. Query speed matters after those properties, not before them.

Limitation: an external collector adds another process, another buffer, and another health signal. On one durable server with modest traffic and an operator who can reliably access it, local files may be the more proportionate choice. In a short-lived compute environment where standard output collection is unavailable, a bounded direct client may be necessary. The trade-off is explicit: the chosen design reduces language-level transport coupling, but it does not remove transport failure or operational ownership.

The alert has two clocks. Soon after the scheduled time, look for import_started; after the documented execution window, look for a terminal event. Do not invent the window from a percentile that nobody has measured. Establish it from the schedule and the service's accepted operating limit, record that limit with the runbook, and revise it through review. This is a compliance-friendly boundary: the alert rule has an owner and a reason, rather than an unexplained number embedded in a dashboard.

Critical path in Go

All emitters should serialize the same small envelope. The following Go example shows the contract rather than a destination SDK; Node.js, FastAPI, and Rails should produce the same field names and semantics through their native structured loggers.

package importlog

import (
    "encoding/json"
    "errors"
    "io"
    "time"
)

type Event struct {
    Timestamp       time.Time `json:"timestamp"`
    SchemaVersion   int       `json:"schema_version"`
    EventName       string    `json:"event_name"`
    EventID         string    `json:"event_id"`
    ExecutionID     string    `json:"execution_id"`
    Service         string    `json:"service"`
    Deployment      string    `json:"deployment_revision"`
    Outcome         string    `json:"outcome,omitempty"`
    RecordsRead     int       `json:"records_read,omitempty"`
    RecordsAccepted int       `json:"records_accepted,omitempty"`
    RecordsRejected int       `json:"records_rejected,omitempty"`
}

func WriteEvent(w io.Writer, e Event) error {
    if e.EventID == "" || e.ExecutionID == "" || e.EventName == "" {
        return errors.New("event_id, execution_id, and event_name are required")
    }
    if e.SchemaVersion != 1 {
        return errors.New("unsupported schema version")
    }
    return json.NewEncoder(w).Encode(e)
}

The critical ordering rule is more important than the serializer: commit the business result, then emit the terminal evidence with the same execution ID. If a crash occurs between those operations, reconciliation must be able to inspect durable business state and repair the missing evidence without repeating the import's business effects. The import itself therefore needs idempotency keys at its write boundary. A log line cannot provide idempotency.

Keep the audit distinction explicit. event_id identifies one emitted fact and supports transport deduplication; execution_id identifies the scheduled attempt and joins its lifecycle; a domain idempotency key protects catalog mutations. Reusing one identifier for all three looks economical until a retry creates a second event for the same safe business operation.

Tiny details matter. A newline-delimited encoder keeps one event per record. Serialization failure is returned to the caller rather than silently discarded. The caller must surface that failure through a separately monitored path, because recursively logging a logger failure merely produces another attempt through the broken path.

Do not retry forever.

Rollout, rollback, and proving the alarm

Start with a versioned schema and dual-read queries, not dual-write destinations inside every service. During rollout, the search rule accepts the current and previous schema versions while new emitters are deployed. Once every supported revision emits the new version and the rollback window closes, retire the old query. This keeps a rollback from turning valid old events into apparent silence.

The deployment gate should exercise five outcomes against a non-production schedule: a normal non-empty result, a valid zero-record result, a worker that never starts, a worker that starts but never finishes, and duplicate delivery of one terminal event. Assert both sides. Successful executions must remain quiet, while missing terminal evidence must open an alert containing the execution ID, schedule, last observed stage, and runbook owner.

Then test the collector boundary by interrupting its destination and restoring it. The acceptance condition is not “no lines were lost” unless the chosen collector and infrastructure have been configured and verified to make that guarantee. A defensible condition names the permitted loss or delay, observes queue pressure, and confirms what happens when the buffer limit is reached. Compliance limits belong in this test plan too: authorized responders can retrieve the required evidence, unauthorized identities cannot, and expired records are handled under the documented retention rule.

Rollback is a data-compatibility exercise. Before deployment, record the last application revision that understands each schema version. Preserve the prior alert query until that revision is outside the rollback window. If a release changes event meaning, introduce a new field or version; never silently reinterpret an existing field. Append-only evolution makes incident timelines auditable.

No result means no proof. Page on that gap.

The rejected direct-client design still has a place

Sending logs directly from each application to a remote endpoint was rejected because it puts destination authentication, batching, retries, and backpressure into three framework runtimes. Those mechanics enlarge the application rollback surface and can compete with import work during a destination outage. The Logback documentation's appender model illustrates how logging destinations can be attached and customized [2], but the existence of an extension point does not remove the operational coupling created by custom destination code.

Direct clients remain valid when the runtime has no collectable output stream, when a platform contract requires an application-level transport, or when the team can own and test identical failure semantics across every language. The burden of proof is concrete: bound memory, define drop behavior, prevent credentials from reaching event payloads, expose queue saturation through an independent signal, and show that a destination outage cannot block marketplace mutations.

For this marketplace, the external collector wins because it narrows the reversible unit. Applications state facts; collection transports them; the searchable store indexes them; reconciliation checks that every scheduled execution reaches one explainable terminal outcome. Those boundaries let a team change storage or roll back a Node.js worker without rewriting the audit contract.

References

1. Google, “Monitoring Distributed Systems,” Site Reliability Engineering: https://sre.google/sre-book/monitoring-distributed-systems/
2. Logback Manual, “Appenders”: https://logback.qos.ch/manual/appenders.html