RainerBarrett4745

Posted on Oct 13

Structured Logging Explained: 3 Production App Signals for JSON Search

#observability #logging #typescript

TL;DR: Choose the logging shape that can reconstruct one marketplace experiment for one tenant cohort from deployment to outcome. For a small team, start with structured JSON at the application boundary, preserve three correlation signals (tenant_id, experiment_id, and trace_id), and benchmark the full ingest-to-query path with your own retention and region constraints. A polished dashboard is useful. A repeatable incident timeline is the decision.

Choice	Time to first useful query	Incident reconstruction	Operating burden	Better fit
Managed indexed search	Usually shortest	Strong when fields are mapped and retained	Low	Small team that wants self-service now
Managed label-based log search	Short	Strong when labels stay low-cardinality and JSON remains searchable	Low	Team already using metrics and dashboards built around labels
Self-hosted search or log store	Longer	Fully controllable	High	Team with data-control requirements and real platform capacity

Recommendation: run a two-day bake-off on managed indexed search and managed label-based search. Keep self-hosting out of the first round unless control of storage, network, or keys is a hard requirement. Score each option on timeline completeness and operator time, not on a demo dashboard or an advertised ingestion rate.

That matrix describes categories, not a winner. Datadog Logs and Elastic Cloud are examples of managed indexed search; Grafana Cloud Logs is an example of a managed service built around Loki's label-oriented model. OpenSearch and Loki can also be operated by your team. Those names establish concrete boundaries, not endorsements. Deployment model, supported regions, retention controls, and field behavior must be verified against current documentation before a production choice.

How should a FastAPI or Express app choose a structured logging stack?

Valid JSON is only the envelope. Incident reconstruction depends on stable semantics across producers, releases, retries, and asynchronous work. If an Express worker emits tenantId, a FastAPI endpoint emits tenant_id, and a queue consumer omits the tenant entirely, the dashboard can search every event and still fail to answer the question.

JSON is the easy part.

The marketplace case makes this painful. Suppose an experiment changes seller ranking for cohort seller-growth-eu. A conversion dip appears after a deployment. The useful investigation is not "show errors." It is: find requests for the affected cohort, identify the experiment variant, connect API work to background ranking jobs, separate expected business rejections from failures, and order the evidence around the deployment.

Three signals carry most of that load. tenant_id identifies the isolation boundary. experiment_id plus a variant identifies the decision being tested. trace_id joins work that crossed process boundaries. Keep a request or job identifier too, but do not pretend it replaces a trace across services. OpenTelemetry defines trace context propagation for that cross-process relationship; its sampling documentation also explains why a trace may not be recorded even when application logs exist.

This is the first trap: sampled traces and unsampled logs have different completeness. If the only copy of the experiment assignment lives on a sampled span, some incident timelines will have a hole by design. Put the assignment in the structured application event that records the business outcome, then add trace correlation when a trace exists.

Small teams also need field discipline. Use a boring event contract. Reserve names, types, and meanings. Reject accidental high-cardinality values as index labels even though those values can remain in the JSON body. Loki's documentation warns that high-cardinality labels create many streams; the same tenant identifier that is valuable as a searchable field may be a bad label when tenant count grows.

Criterion 1: Can it rebuild the timeline?

Test reconstruction with a fixed incident packet, not free-form clicking. Seed a staging environment with one deployment marker, two tenant cohorts, two experiment variants, a synchronous request, an asynchronous job, a retry, and one business rejection. Then hand another engineer only the question and a time window.

Walk the packet as a timeline. At 10:00 UTC, revision build-1842 is deployed. At 10:04, a request for tenant_7f3a assigns seller-ranking-v3 to treatment and enqueues ranking work. The first job attempt times out, the second succeeds, and the final outcome is accepted. The investigator should be able to start from any one of those facts and reach the others without copying identifiers between three dashboards. If the deployment marker has no immutable revision, the code change cannot be pinned down. If the queue drops trace context, job_id must still connect the attempts, while the originating event preserves the experiment assignment. If the retry omits attempt, duplicate work looks like duplicate customer intent. This staged story does not claim a benchmark result; it defines what a complete result must contain. It also makes disagreements visible. One option may return the first matching line faster while another produces the full sequence with less query glue. Record both.

Test the joins.

Measure four things: time to the first correct query, time to a complete ordered timeline, the number of missing joins, and the amount of query-specific configuration added. The first two are elapsed durations from the exercise. The others are counts. Do not publish universal numbers from this test; the result belongs to your event volume, schema, network path, and team.

A system fails this criterion if an engineer must know which service renamed a field last month. It also fails if a retry looks like a second customer action, or if clock skew silently reverses cause and effect. Emit an event timestamp in UTC, retain the ingestion timestamp, and preserve attempt numbers. During reconstruction, sort primarily by event time but display ingestion delay. Late arrival is evidence.

Deployment markers deserve the same schema treatment as application events. Record the service, immutable revision, environment, and deployment time. Avoid stuffing the full build manifest into every log line. One compact marker can anchor the timeline, while service.version on application events identifies which code produced them. The OpenTelemetry semantic conventions define common attribute names, including service resource attributes, and reduce translation glue between runtimes.

No magic here. A dashboard built on inconsistent events makes inconsistency easier to chart.

Retries lie when attempts are invisible.

Criterion 2: Will the team keep it operable?

The easiest stack is the one that stays understandable after the author goes offline. Count every configuration surface: application serializer, collector, routing rules, index or label policy, retention, access control, saved queries, alerts, and dashboard definitions. More knobs can buy control, but each knob needs an owner and a test.

Configuration is work.

Benchmark the hot path and the human path separately. For the hot path, measure application overhead, dropped-event behavior, batching, queue pressure, and end-to-end arrival latency under representative load. For the human path, ask a developer who did not build the pipeline to locate the failed cohort and share a stable query. Capture their dead ends. That is the DX benchmark I care about because glue work shows up there immediately.

Regional operation is a gate, not a footnote. A US/EU marketplace should document where log data is accepted, processed, stored, backed up, and queried; which roles can cross regional boundaries; and how deletion and retention are enforced. Vendor deployment regions and terms change, so verify them during procurement rather than freezing them into an architecture diagram. For self-hosting, the same questions still exist. Your team owns the answers.

Regions are architecture.

Do a failure drill before selection. Block the destination briefly in staging and confirm the application keeps serving, buffers stay bounded, and the logger exposes drops without recursively logging its own failure. Then restore the path and inspect ordering and duplication. Logging must degrade independently from the request path.

Cost belongs in the test, but it is not the thesis. Model bytes after serialization, indexing or label choices, retention tiers, query scanning, network transfer, and the engineering time needed to operate the pipeline. Use a representative day and an incident-heavy day. A low ingestion quote cannot rescue a stack that takes hours to reconstruct an incident.

A minimal event contract in TypeScript

One schema shared by Express middleware, queue producers, and test fixtures is more valuable than framework-specific logging tricks. FastAPI can emit the same wire shape from Python, but this article keeps all code in TypeScript so the contract is easy to scan.

interface MarketplaceEvent {
  timestamp: string;
  severity: "info" | "warn" | "error";
  event_name: "experiment_outcome" | "job_attempt" | "deployment";
  service_name: string;
  service_version: string;
  tenant_id: string;
  experiment_id?: string;
  experiment_variant?: string;
  trace_id?: string;
  request_id?: string;
  job_id?: string;
  attempt?: number;
  outcome?: "accepted" | "rejected" | "failed";
  duration_ms?: number;
}

function writeEvent(event: MarketplaceEvent): void {
  const checked: MarketplaceEvent = {
    ...event,
    timestamp: new Date(event.timestamp).toISOString(),
  };

  process.stdout.write(`${JSON.stringify(checked)}\n`);
}

writeEvent({
  timestamp: new Date().toISOString(),
  severity: "info",
  event_name: "experiment_outcome",
  service_name: "ranking-api",
  service_version: "build-1842",
  tenant_id: "tenant_7f3a",
  experiment_id: "seller-ranking-v3",
  experiment_variant: "treatment",
  trace_id: "4bf92f3577b34da6a3ce929d0e0e4736",
  request_id: "req_91c2",
  outcome: "accepted",
  duration_ms: 87,
});

The trace_id example uses the 32-lowercase-hex-character form shown by the W3C Trace Context specification. In production, copy it from validated trace context rather than generating an unrelated value in the logger. Also treat identifiers as potentially sensitive: define redaction before serialization, test it with fixtures, and never log secrets, authorization headers, or raw marketplace payloads.

This sample writes to standard output because it keeps transport out of application code. A process supervisor or collector can handle batching and delivery. The deliberate limitation is equally important: writeEvent is not a production queue, retry system, or schema validator. Add those at the boundary that owns them, and benchmark the added failure modes.

Schema tests should reject missing required fields and changed types. Add a golden fixture for each event name, then run the same reconstruction queries against staged data before a schema migration ships. Version the contract when semantics change; do not version it because a field was added compatibly. Config bloat starts when every service invents an exception.

When is the runner-up the better choice?

Choose the label-based runner-up when the team already understands its query model, runs shared dashboards for logs and metrics, and can enforce a tiny label set while leaving cohort and trace identifiers in parsed JSON. It can be the cleaner operational fit. The bake-off still has to prove that developers can find a single tenant cohort without promoting unbounded identifiers into labels.

Choose indexed search instead when ad hoc field exploration is the dominant incident workflow and schema variation is unavoidable during a migration. The trade-off is greater attention to mappings, field explosion, and index lifecycle behavior. Elastic's documentation describes limits intended to prevent mapping explosion; test your evolving payloads against those limits rather than assuming every new JSON key is free.

Self-hosting becomes rational when control requirements are explicit and the team can staff upgrades, capacity planning, backup recovery, access control, and query-performance work. "We already run containers" is not that capability. Run a restore drill and an upgrade rehearsal before calling this option easy.

The final decision record should fit on one page: the incident packet, measured reconstruction results, missing joins, regional-data answers, failure-drill outcome, estimated volume model, and the reason the runner-up lost. Re-run it after a material schema, traffic, or compliance change. Pick the system that preserves the evidence your team needs and asks for the least unexplained glue.

DEV Community