FinnOakley52947

Posted on Oct 17

12 Structured JSON Events to Vet a Small SaaS Logging Service for Rollbacks

#observability #logging #experimentation

TL;DR: For a small US/EU gaming SaaS, the best simple logging setup is the one that can put an experiment into one of three states: continue, rollback, or hold. Use structured JSON outcome events, keep tenant cohort and deployment identity on every event, and make missing evidence produce hold instead of a reassuring ratio. Twelve fixture events are enough to prove the contract before debating storage features.

This changes the buying question. Fast ingestion and a tidy search screen matter, but they do not make a rollback safe. A pipeline must preserve the fields and timing semantics that the rollback rule consumes. Otherwise the operator is looking at activity, not evidence.

What should a small SaaS app logging service prove?

Imagine a matchmaking experiment running for two tenant cohorts. standard tenants stay on the baseline. early_access tenants receive a candidate deployment. Traffic is served in the US and EU, and each completed allocation writes one JSON event.

The tempting query is failures divided by attempts. It is also incomplete. If EU candidate events arrive late, if a retry creates two records for one operation, or if an older Node.js process omits cohort, the ratio can still render. Nothing about a plausible decimal proves that the compared sets are complete. Express middleware can attach request context, and PostgreSQL can expose database activity, but neither layer should be assumed to know the final experiment outcome unless the app records it deliberately.

The fix is a decision contract. Before rollout, define the required slices and the conditions that prevent a decision. For this example, both variants must have valid events in both regions, every event must carry the expected schema version, and duplicate event IDs must be visible. A missing required slice returns hold. It does not become zero.

That last rule matters most. Unknown is a state, not a number.

No guesswork.

OpenTelemetry's logs data model distinguishes the time when an event occurred from the time when it was observed. That distinction fits this job: use event time to place an outcome in the experiment window, while observed time can reveal delivery delay. If a logging path cannot preserve both concepts, document which timestamp survives and test the boundary behavior. Do not let ingestion time silently redefine the canary window.

Build the rollback interlock before choosing storage

The smallest implementation is an application-owned outcome event plus a pure decision function. The event belongs at the point where the application knows what the player actually received. A database log cannot supply experiment assignment or the final response by itself.

This TypeScript example is deliberately boring. It emits newline-delimited JSON to standard output, then evaluates already parsed events. The threshold of 0.05 is an illustrative product policy, not a universal recommendation.

type Region = "us" | "eu";
type Cohort = "standard" | "early_access";
type Variant = "baseline" | "candidate";
type Outcome = "allocated" | "rejected" | "timeout";

type AllocationEvent = {
  schemaVersion: 1;
  eventName: "match.allocation.completed";
  eventId: string;
  operationId: string;
  eventTime: string;
  observedTime: string;
  tenantKey: string;
  cohort: Cohort;
  experiment: "matchmaking_canary";
  variant: Variant;
  deployment: string;
  region: Region;
  outcome: Outcome;
};

type Decision =
  | { state: "continue"; candidateFailureRate: number }
  | { state: "rollback"; candidateFailureRate: number }
  | { state: "hold"; reason: string };

export function writeAllocationEvent(event: AllocationEvent): void {
  process.stdout.write(`${JSON.stringify(event)}\n`);
}

export function decide(events: AllocationEvent[]): Decision {
  const unique = new Map<string, AllocationEvent>();

  for (const event of events) {
    if (unique.has(event.eventId)) {
      return { state: "hold", reason: `duplicate eventId: ${event.eventId}` };
    }
    unique.set(event.eventId, event);
  }

  const required = [
    "baseline:us",
    "baseline:eu",
    "candidate:us",
    "candidate:eu",
  ];
  const present = new Set(
    [...unique.values()].map((event) => `${event.variant}:${event.region}`),
  );
  const missing = required.filter((slice) => !present.has(slice));

  if (missing.length > 0) {
    return { state: "hold", reason: `missing slices: ${missing.join(", ")}` };
  }

  const candidate = [...unique.values()].filter(
    (event) => event.variant === "candidate",
  );
  const failures = candidate.filter(
    (event) => event.outcome !== "allocated",
  ).length;
  const candidateFailureRate = failures / candidate.length;

  return candidateFailureRate > 0.05
    ? { state: "rollback", candidateFailureRate }
    : { state: "continue", candidateFailureRate };
}

There is no destination SDK in that code. Good. The transport can change without rewriting the decision vocabulary. There is also no attempt to hide duplicate records with a convenient Set and move on. A duplicate blocks the decision because the sample's integrity is now unresolved.

The function is intentionally stricter than many dashboards. It proves a useful point, though it is not production-complete: a real policy must also bind an experiment window, validate timestamps and schema at ingestion, check that expected deployments are represented, and state how retries map to operations. Those rules depend on the application's rollout protocol.

The 12-event test catches the dangerous kind of simple

Create a fixture with 12 valid events: three outcomes for each variant-region pair. Keep the cohort, deployment, event time, observed time, and operation ID explicit. This is a contract fixture, not a traffic benchmark and not evidence that 12 production events are statistically sufficient.

Run four deterministic mutations against it:

Remove all candidate:eu events. The result must be hold.
Repeat one eventId. The result must be hold.
Change one candidate outcome from allocated to timeout. The computed rate must change according to the declared policy.
Move one observedTime later without changing eventTime. Its experiment-window membership must stay unchanged.

This is where simple services separate themselves from simplistic ones. Search isn't the benchmark. The benchmark is whether the exported records let the same pure function return the same state.

Measure two clocks during setup: time to the first searchable event and time to the first reproducible decision. The first exposes integration friction. The second includes export, timestamp behavior, JSON type preservation, and enough query expressiveness to retrieve a bounded experiment window. I care more about the second clock because it measures the whole job, but recording both shows where the glue accumulated.

Avoid transforms until the fixture forces one. Parser rules, field renames, routing filters, and derived columns all become configuration that can drift away from the TypeScript event type. Each transform needs a contract assertion. If it has none, delete it. This is a blunt DX trade-off: a collector rule may remove repeated app code, but it also moves behavior away from the TypeScript compiler and into a separately deployed configuration surface. The extra layer earns its place only when one tested policy can replace several copies of application glue.

Error grouping deserves a separate check. Sentry documents that grouping can be based on event information such as stack traces and can be altered with fingerprints. That is useful for bringing related errors together. It does not provide the denominator of completed match allocations, and it cannot recover cohort context that the application never emitted. Keep grouped exceptions for diagnosis; keep canonical outcome events for the rollback decision.

Choose a path by counting failure surfaces

Once the interlock passes locally, compare architecture shapes rather than brand checklists. For a single deployment path, JSON on standard output with runtime shipping has one small application integration. Its boundary is operational: the team must verify how the runtime handles backpressure, multiline output, restarts, and shutdown.

A local collector adds a process and configuration, but it can centralize batching, redaction, regional routing, and export when several services need the same policy. A direct destination SDK moves buffering and delivery behavior into the application lifecycle. That can be acceptable for a narrow stack, yet it ties shutdown and failure handling to destination-specific code.

Path	App integration	Useful when	Main limitation
JSON to standard output	One typed writer	One runtime owns log shipping	Runtime buffering and shutdown behavior need verification
JSON through a local collector	Writer plus collector policy	Several services share routing or redaction	Another process and config surface can drift
Direct destination SDK	SDK lifecycle in the app	Destination-specific behavior is required	Delivery and migration logic enter application code

The limitation of the 12-event fixture is just as important: it tests record integrity and decision semantics, not statistical power, sustained throughput, or long-retention query latency. A team still needs workload-sized tests for those properties. Likewise, standard output isn't suitable when the runtime cannot provide the delivery guarantees in the operating contract; a collector isn't suitable when nobody owns its lifecycle; and a direct SDK is a poor fit when portability is the dominant constraint.

There is no abstract winner. Count the places that can alter or lose a decision field. Then force each one to fail.

For US/EU operation, regional selection is only one input. The team still has to define where records are processed, who can read them, how long raw identifiers remain available, and where cross-region cohort aggregates may be computed. The application should avoid secrets, authorization headers, player names, email addresses, and database connection strings in logs. A pseudonymous tenant key may be operationally useful, but its handling still belongs in the system's data policy.

PostgreSQL telemetry also stays in its lane. Database activity helps explain query and transaction behavior. The application owns whether a matchmaking request produced allocated, rejected, or timeout, along with the experiment and cohort context. Correlate the layers with an operation or trace identifier; do not infer a player-visible outcome solely from database success.

Cost comes after correctness. Estimate it from the actual fixture-expanded event shape, expected event volume, retention window, indexing policy, and export needs. Avoid comparing headline entry prices. They change, and they say little about the operational bill created by high-cardinality fields or duplicated ingestion.

Cheap is not simple.

What I would change when the canary outgrows one service

At larger scale, I would keep the three-state contract and change its inputs. A collector can validate and route events. Bounded metrics can provide fast detection by region, variant, cohort, outcome, and deployment, while access-controlled logs retain operation IDs for investigation. Tenant and operation IDs should not become unrestricted metric labels because their value sets grow with tenants and requests.

Schema coordination grows too. One producer can keep its TypeScript type and fixture together. Independently deployed producers need compatibility checks so old and new records can coexist during rollout. Additive fields are easier to manage than silent renames, but the decision function still needs tests for both versions.

I would also separate detection from proof. Metrics can wake an operator quickly. The bounded event set explains which cohort, deployment, and operations produced the signal. Neither should quietly compensate for missing data in the other.

The final selection rule stays compact: choose the least-configured path that preserves the rollback contract under loss, delay, duplication, and regional routing. A setup that emits a fast first event but cannot distinguish hold from continue has optimized the wrong milestone.

DEV Community