Small SaaS Evidence: Cheap Centralized Node.js Structured Logging Retention

#observability #logging #fintech

Short answer: centralize structured application logs, but index only the fields needed for immediate triage and retain a separate, access-controlled evidence stream for reconstruction. For a small fintech SaaS comparing an experiment across tenant cohorts, the dominant billable term is usually the volume sent into the searchable tier multiplied by its retention period; the least complex useful change is therefore to stop treating every event as equally searchable. Keep a compact incident index, archive the fuller approved record, and make tenant_id, cohort, experiment, job, run_id, event_id, and occurred_at consistent across APIs, container workers, and cron jobs.

This separates cost from correctness. A log search helps reconstruct an incident, but it is not a ledger, and it cannot establish exactly-once processing by itself. The ledger and idempotency store remain authoritative; logs explain how a request moved through the system. Datadog's public pricing page separates log ingestion from indexed retention, illustrating why reducing indexed volume can matter even while collection continues. Exact commercial terms change, so architecture should depend on measured bytes and retention requirements, not a quoted unit price.

How can a small SaaS keep centralized logging cheap?

Start with bytes. For each source class, measure events per run, average encoded size, runs per day, and retention days. Then distinguish collected, indexed, and archived bytes because they have different operational value.

Count first.

Source	Immediate search value	Reconstruction value	Treatment
API outcome	High	High	Index selected fields; archive approved event
Ledger posting result	High	High, but not authoritative	Index identifiers and outcome
Cron progress	Low	Medium	Aggregate or sample
Cron completion	High	High	Index
Debug detail	Low outside incidents	Variable	Disable by default or retain briefly

Use a worksheet rather than calling the volume “small.” If a hypothetical reconciliation run emits 40,000 progress events and one completion event, progress dominates event count even though completion usually answers the first operational question. This is an illustrative capacity-model input, not a benchmark. Measure the real encoded payload after enrichment; container metadata and repeated exception text consume bytes too.

The change that moves the dominant term is reducing what enters long-lived search: emit summaries, remove redundant fields, cap exception payloads, and route low-value detail to short retention or archives. The same envelope can be emitted as JSON by a Node.js service, a Docker worker, or a cron job even though the illustrative encoder below is Go. This centralized design can serve self-service investigation without placing every message in the expensive search tier. Do not shorten all retention indiscriminately. That can produce a smaller bill and an unreconstructable settlement dispute.

Which fields reconstruct a cohort incident?

The schema must connect a customer-visible outcome to an experiment assignment and every asynchronous attempt. A useful event is a claim with stable join identifiers. Free-form prose is secondary.

package eventlog

import (
    "encoding/json"
    "errors"
    "io"
    "strings"
    "time"
)

type Event struct {
    SchemaVersion  int       `json:"schema_version"`
    EventID        string    `json:"event_id"`
    OccurredAt     time.Time `json:"occurred_at"`
    TenantID       string    `json:"tenant_id"`
    Cohort         string    `json:"cohort"`
    Experiment     string    `json:"experiment"`
    Job            string    `json:"job"`
    RunID          string    `json:"run_id"`
    IdempotencyKey string    `json:"idempotency_key"`
    Attempt        int       `json:"attempt"`
    Outcome        string    `json:"outcome"`
}

func WriteEvent(w io.Writer, e Event) error {
    if strings.ContainsAny(e.TenantID, "\r\n") {
        return errors.New("tenant_id contains a line break")
    }
    enc := json.NewEncoder(w)
    enc.SetEscapeHTML(true)
    return enc.Encode(e)
}

OWASP advises excluding or masking access tokens, passwords, sensitive personal data, and payment cardholder data, and sanitizing event data against log injection. The narrow envelope therefore excludes request bodies and financial credentials.

A unique event_id makes duplicate delivery visible; it does not prevent duplicate financial effects. Check the idempotency key transactionally where the effect commits, while attempt records retries and run_id groups one scheduled execution. Two entries may describe one committed effect, two attempts, or duplicated shipment. Search results cannot decide which occurred.

Keep cohort assignment stable for the business operation. If assignment changes mid-workflow, record the value used at decision time instead of deriving it from current tenant state. The audit join then connects tenant, assignment, operation, attempt, and authoritative ledger result.

How much retention is enough for an investigation?

There is no defensible universal number in the cited evidence. Set searchable and archival windows from the longest plausible discovery delay, investigation needs, and applicable legal requirements. Document the owner and review date. Compliance constrains the design; it is not a decorative reason to retain everything forever.

A practical policy has three clocks: hot search for on-call triage, archive for delayed reconciliation and approved audit needs, and deletion for the organization's retention obligations. Qualified counsel and the responsible compliance owner must define jurisdiction-specific limits.

Attach retention to event classes, not the logging account as a whole. Authentication failures, payment outcomes, health checks, and verbose progress messages have different evidentiary value and sensitivity. A short searchable window can coexist with a longer approved archive when restoration is tested, access is logged, and deletion covers governed replicas and exports.

Make self-service queries safe and repeatable

Self-service means an engineer can answer a bounded question without unrestricted access to every tenant. Require tenant scope, cohort or experiment, and a time interval. Return identifiers and operational fields by default; sensitive details need separate authorization.

Begin with counts grouped by cohort, job, and outcome, then inspect relevant run_id values. Compare rates only after joining an independently defined denominator because logs lost in transit could make a failing cohort look healthy. Reconcile suspicious runs against authoritative idempotency and ledger records.

Fast search helps. It is not proof.

There is a hard limitation: this two-tier pattern is not suitable when investigators must run low-latency, ad hoc searches across every raw event for the entire mandated retention window. In that case, retain and index the required event classes, accept the larger capacity requirement, and control access tightly. The opposite trade-off also matters. Teams that cannot operate archive restoration reliably should choose a simpler single-tier design until they can test recovery; a cheap archive that cannot be searched under incident pressure is not evidence in practice.

Treat the schema like an API: version it, reject unexpected high-cardinality fields, and test representative events during deployment. Confirm that API containers and cron jobs emit valid JSON and propagate the same identifiers. Monitor pipeline health separately from application outcomes so silence cannot masquerade as success.

Test archive restoration on a schedule and record who requested it, what range was retrieved, and when the temporary working set was deleted. An archive never restored is an assumption. During an incident, that assumption is expensive.

The deliberate loss

Stop keeping repetitive progress events in long-lived search, exclude raw payloads and credentials entirely, and delete archived classes when their approved window ends. Retain compact completion records, stable correlation fields, pipeline-health evidence, and authoritative transaction records outside logging.

The cost is plain: after detail expires, an investigator may know that a run failed and which financial operations committed, yet be unable to replay every internal branch. Preserve detail where that branch history has defined incident, audit, or reconciliation value. Otherwise the organization pays continuously for hypothetical evidence while increasing sensitive data exposure.

Optimize indexed retention only after the reconstruction contract is explicit. For tenant-cohort experiments, name the questions a reviewer must answer, authoritative systems used to verify effects, permitted fields, and maximum discovery delay. The result stays searchable without pretending an index is a ledger.

DEV Community