Production API Log Management: 5 JSON Checks for Tenant Experiments

The useful outcome is a timeline that explains which tenant cohort was assigned, when its scheduled comparison ran, and why a request or job failed. The least complex option is structured application logging with a stable run identifier, centralized JSON search, and one dashboard tied to an alert. Start there before comparing storage engines or interface features.

TL;DR: The best log management choice for a production Express API is the one that passes an incident-reconstruction drill with your own data. Test five things: correlation across requests and scheduled jobs, explicit outcome events, fast fielded search, controlled sensitive-data handling, and a dashboard that leads from an alert to evidence. A polished error chart cannot compensate for a missing cohort assignment or duplicate run.

Imagine the page: tenant_experiment_completion is below its expected threshold for the 02:00 UTC window. On-call opens the linked dashboard and sees 94 scheduled cohort runs, 91 terminal outcomes, two retries, and one run with no terminal event. Those numbers are example test data, not a universal threshold. The operational question is immediate: did the comparison fail, arrive late, run twice, or merely stop emitting evidence?

That is the selection test. Can the system answer it without joining screenshots, shell history, and somebody's memory?

1. What Should Production API Log Management Reveal in JSON Logs?

An HTTP request ID is too narrow for this property-management workflow. A tenant manager may configure an experiment in one request, a scheduler may enqueue cohort work hours later, and workers may process the control and treatment groups independently. Preserve request_id, but add stable identifiers for experiment_id, tenant_id, cohort_id, and run_id. A retry keeps the same run ID and increments attempt; a genuinely new scheduled run gets a new run ID.

Do not use an email address, resident name, or street address as a correlation key. Opaque identifiers are easier to search consistently and reduce the chance that an operational query exposes personal data.

The schema below is deliberately small. It records a state transition rather than dumping an entire request body:

package logging

import (
    "encoding/json"
    "io"
    "time"
)

type RunEvent struct {
    Timestamp    time.Time `json:"timestamp"`
    Event        string    `json:"event"`
    TenantID     string    `json:"tenant_id"`
    ExperimentID string    `json:"experiment_id"`
    CohortID     string    `json:"cohort_id"`
    RunID        string    `json:"run_id"`
    RequestID    string    `json:"request_id,omitempty"`
    Attempt      int       `json:"attempt"`
    Outcome      string    `json:"outcome,omitempty"`
    DurationMS   int64     `json:"duration_ms,omitempty"`
}

func WriteRunEvent(w io.Writer, e RunEvent) error {
    return json.NewEncoder(w).Encode(e)
}

The idempotency reflex matters here. I've been paged for both missed jobs and duplicate deliveries; a timeline that hides the distinction makes either page harder to resolve. Searching for run_id should reveal whether attempt 2 replaced a failed attempt, or whether both attempts committed results. If the logs cannot distinguish an attempted delivery from a committed cohort result, they cannot establish duplicate impact.

2. Record outcomes, not hopeful activity

The page above fired late because completion was the first trustworthy signal. Work backward. A scheduler log saying "queued" proves only that enqueueing was attempted; a worker log saying "started" proves only that execution began. For each run, emit a bounded set of events such as scheduled, started, and one terminal outcome: completed, failed, or cancelled. Define those terms in the runbook.

Short gaps count.

The missing terminal event should have produced an earlier signal. Derive a metric from the state transitions or instrument the same code path with a counter and latency histogram. OpenTelemetry describes a counter as a value that accumulates over time and a histogram as a distribution of measurements; those semantics fit completed runs and run duration, respectively. Logs retain the event detail needed for reconstruction, while metrics make threshold evaluation efficient.

This separation prevents a common category error: treating a growing count of error lines as the service-level signal. One failed attempt followed by a successful retry may generate an error log but still produce a correct experiment result. Conversely, a silent stalled run may produce no error line at all.

3. Test JSON search with incident-shaped questions

Beginner-friendly search should still be fielded search. During an evaluation, load synthetic events for multiple tenants and cohorts, including one late completion, one explicit failure, and one duplicated attempt. Then time how long it takes an engineer unfamiliar with the interface to answer these questions:

1. Which scheduled runs lack a terminal outcome for the window?
2. Did both attempts for one run_id commit a result?
3. Which cohort produced the error, and what request originally configured it?
4. Did the problem affect one tenant or cross a service boundary?

Exact syntax will differ across systems, so the durable requirement is support for equality filters, time ranges, existence checks, grouping, and a way to inspect the surrounding events without losing the current query. Save the reconstruction query alongside the alert or runbook. Do not rely on a dashboard author remembering it at 03:00.

This method has a limitation: structured logs can reconstruct emitted events, but they cannot prove what happened in code paths that never emitted one. The trade-off is deliberate instrumentation work and schema discipline in exchange for a queryable timeline. For a tiny service with low volume and no asynchronous work, local JSON files plus controlled rotation may be sufficient; centralized search becomes valuable when incidents cross processes, hosts, or retention boundaries.

Search latency also needs a local acceptance target. Pick it from the incident workflow and retained volume rather than a marketing benchmark. A reasonable drill might require the first scoped result before the responder abandons the query and reaches for another tool, but the actual number belongs to the team operating the service. Record the dataset size, query, time range, and concurrency whenever comparing candidates; otherwise the timing has no diagnostic value.

4. Is the dashboard an evidence trail or a wall chart?

A useful first panel shows scheduled runs versus terminal outcomes by time window. The next splits missing or failed outcomes by cohort. A link then carries the exact run_id, tenant_id, and time range into logs. This is an alert-to-action trace, not a collection of attractive charts.

Keep request rate and HTTP error rate nearby, but do not let them stand in for job correctness. The API can return successful configuration responses while the later cohort comparison never completes. Likewise, an experiment summary can be wrong after a duplicate commit even if every worker returned success.

Instrument the boundary where the result becomes durable. Emit the terminal log event and increment the completion metric only after that boundary reports success. If those operations cannot be atomic, include enough identity to reconcile them and document the ordering. The runbook should state which signal is authoritative.

During a game day, remove one terminal event from the synthetic stream. The alert should open the dashboard, the dashboard should isolate the cohort, and the linked query should expose the last known transition. Also inject a retry that succeeds. On-call should see one logical completion, not interpret two attempts as two completed experiments.

The dashboard passes only if it shortens the path to a falsifiable incident narrative. More panels are not a substitute.

5. Put retention, erasure, and alert noise in the decision record

Production request logs tempt teams to capture everything "for later." That creates two problems: secrets and personal data enter an operational store, and the volume obscures the few fields needed for reconstruction. Allowlist fields at the application boundary. Redact authorization values and session material before serialization, and avoid request or response bodies by default. Test redaction with fixtures rather than trusting a visual sample. This means giving up some ad hoc payload debugging; the reason is simple: an operational convenience does not justify retaining unrestricted tenant data.

Retention is a workload decision. The searchable window must cover the longest useful incident and experiment-analysis horizon, while archive and deletion behavior must match organizational obligations. GDPR Article 17 establishes a right to erasure in specified circumstances and also lists exceptions; legal and privacy owners must define how that applies to identifiers and derived records. An engineering team should verify deletion behavior, access controls, and audit evidence instead of inferring compliance from a storage setting.

Finally, tune the alert against expected scheduling behavior. A threshold of fewer than 94 completions from 94 schedules would page on one missing outcome in the example above, but it might also page while a valid slow run remains inside its deadline. Add a lateness allowance grounded in the job's completion objective, evaluate by tenant or cohort only where that split is actionable, and test maintenance windows. Too loose misses the incident. Too tight trains responders to ignore the page.

The decision record can stay compact:

Check	Evidence to capture	Failure condition
Correlation	One query for a full run_id timeline	Requests, attempts, or commits cannot be joined
Outcomes	Exactly one logical terminal result	Started work disappears or retries look like successes
Search	Saved incident questions over test volume	Required fields cannot be filtered or grouped
Handling	Redaction and deletion test results	Sensitive fields leak or governed records persist
Alert path	Game-day alert, dashboard, and linked query	Page cannot isolate a concrete missing outcome

Choose only after the same drill runs against every candidate with the same schema and dataset. Operating effort, ingestion limits, query behavior, access control, retention, and exportability all belong in the record. Price can be a constraint, but it does not repair an unreconstructable incident.

The final check is uncomfortable and useful: deliberately move the completion threshold until it produces a false page. Measure what the responder sees and how quickly the page can be disproved. That cost belongs beside detection coverage, because an alerting design that repeatedly interrupts on-call without identifying a tenant, cohort, and run is another production failure mode.

Further reading

• OpenTelemetry, "Metrics signal concepts": https://opentelemetry.io/docs/concepts/signals/metrics/
• GDPR Article 17, "Right to erasure": https://gdpr-info.eu/art-17-gdpr/