DEV Community

LangstonHughes2689
LangstonHughes2689

Posted on

Debug Published Gaming Events Never Seen When Event Names Drift

A gaming device can be online, publishing, and still look frozen on a live dashboard. Treat that contradiction as a naming-contract failure until the evidence clears it. Capture the exact event name at emission and reception, validate it against one shared runtime set, and preserve a cursor so reconnect tests cannot hide the original gap. A transport health check alone answers the wrong question.

TL;DR: instrument the event's identity before inspecting reconnect code. A typo such as device.statsu can travel through a healthy channel while every consumer waits for device.status. Reject unsupported names at the producer boundary, count them, and test live delivery separately from backfill.

How should I debug published events that are never seen?

A connection and an application event are different layers. The W3C WebRTC specification defines data channels and their transport behavior; it does not define a gaming dashboard's event vocabulary. device.status, match.presence, and controller.battery are application contracts. The transport cannot infer that one string is misspelled.

That distinction sets the first debugging move. Do not begin with reconnect timers, charts, or a wider retry loop. Put one correlation ID on a single device update, then inspect four facts: the name the producer attempted, whether runtime validation accepted it, the cursor assigned to the accepted record, and the name the dashboard subscribed to. If the first and fourth strings differ, no amount of reconnect tuning repairs the contract.

The nasty version is syntactically valid. JSON parses. Authentication succeeds. The connection remains open. Yet the dashboard handler never runs because dispatch uses a different key. Silence looks like packet loss, so engineers chase the network. A concrete probe makes that detour avoidable: publish exactly 1 device.status update for console-42, carry 1 correlation ID through validation and append, and require the dashboard trace to show that same name and ID. If validation reports device.statsu, stop there. There is no useful reconnect experiment yet. If append returns a cursor but the receiver has no matching trace, move one boundary forward and inspect subscription dispatch. This is slower to describe than to run, which is precisely what I want from incident tooling.

One event. Four checkpoints.

I would benchmark this path with one boring number: time from a known emission attempt to a conclusive classification. The useful result is a bounded answer: rejected name, accepted live event, missing cursor, or replayed event. If the tooling only says connected, it has measured almost nothing about the bug.

Build the smallest runtime contract

Keep supported names in one tiny module. Avoid a second list in dashboard code and a third list in deployment configuration. That is config bloat wearing a safety badge.

No alias maze.

export const eventNames = [
  "device.status",
  "controller.battery",
  "match.presence",
] as const;

export type EventName = (typeof eventNames)[number];
const supportedNames = new Set<string>(eventNames);

export function parseEventName(value: string): EventName {
  if (!supportedNames.has(value)) {
    throw new Error(`unsupported event name: ${JSON.stringify(value)}`);
  }
  return value as EventName;
}
Enter fullscreen mode Exit fullscreen mode

The tuple helps TypeScript callers. The set checks values arriving from configuration, an API, a queue, or another language. Compile-time typing cannot protect a raw string crossing a runtime boundary. That is the hole worth closing.

Make the publish boundary produce evidence before it touches the transport. Keep the interface generic and small.

type DeviceStatus = {
  deviceId: string;
  state: "online" | "idle" | "offline";
  observedAt: string;
};

type AcceptedEvent = {
  id: string;
  cursor: string;
  name: "device.status";
  payload: DeviceStatus;
};

interface EventStore {
  append(
    name: "device.status",
    payload: DeviceStatus,
    correlationId: string,
  ): Promise<AcceptedEvent>;
}

export async function publishDeviceStatus(
  store: EventStore,
  rawName: string,
  payload: DeviceStatus,
  correlationId: string,
): Promise<AcceptedEvent> {
  const name = parseEventName(rawName);
  if (name !== "device.status") {
    throw new Error(`wrong payload contract for event: ${name}`);
  }
  return store.append(name, payload, correlationId);
}
Enter fullscreen mode Exit fullscreen mode

Do not automatically correct device.statsu. That makes a broken deployment appear healthy and leaves two spellings alive in logs or stored history. Fail the attempted publish, expose the rejected literal, and fix the caller.

Also avoid logging the entire payload by reflex. The high-value fields here are correlation ID, event name, device ID, acceptance result, cursor, and timestamps at defined boundaries. A status payload may grow. Debugging should not quietly become uncontrolled data collection.

Separate live delivery from reconnect and backfill

Once the name is proven correct, run two experiments. First, connect a dashboard client and publish one new status. Second, disconnect it, append several accepted updates, reconnect with the last acknowledged cursor, and inspect the ordered results. Combining the paths creates ambiguity: an old event visible after reconnect might have arrived live, been replayed, or been rendered twice.

Evidence Classification Next check
Name rejected before append Contract mismatch Producer input and deployed contract
Accepted cursor exists, no live receive Live dispatch gap Subscription name and correlation
Live receive works, replay omits later cursors Backfill gap Resume cursor and replay range
Replay returns an acknowledged event Duplicate delivery Idempotency and acknowledgment timing

A cursor belongs to an accepted event, not an attempted one. When a typo is rejected, there should be no phantom position for the dashboard to resume past. If a system accepts arbitrary names into one log, the cursor proves persistence but not routability. Record both the accepted name and cursor.

The consumer should make duplicates harmless because reconnect can overlap the last rendered event and the last acknowledged event. Use the accepted event ID as the deduplication key. Advance the stored cursor only after applying the status.

type DashboardState = {
  devices: Map<string, DeviceStatus>;
  seenIds: Set<string>;
  cursor: string | null;
};

export function applyStatusEvent(
  state: DashboardState,
  event: AcceptedEvent,
): DashboardState {
  if (state.seenIds.has(event.id)) return state;

  const devices = new Map(state.devices);
  devices.set(event.payload.deviceId, event.payload);
  const seenIds = new Set(state.seenIds);
  seenIds.add(event.id);
  return { devices, seenIds, cursor: event.cursor };
}
Enter fullscreen mode Exit fullscreen mode

Tiny code. Useful boundary.

Include a deliberate typo in the reconnect test, but keep that rejected attempt outside the expected replay sequence. Append valid statuses for two device IDs and reconnect from the earlier valid cursor. Assert event IDs, names, and final states. A count-only assertion can pass when the wrong event type fills the slot.

Keep the incident observable

Three counters and one structured trace are enough to start. Count attempted publishes by result, rejected names by a bounded reason, and accepted events observed by the dashboard path. Trace a sampled correlation ID across validation, append, live dispatch, reconnect request, replay, and render. Do not use raw misspellings as metric labels; keep them in a sampled log field where they cannot expand metric cardinality.

This approach has a real limitation. A shared runtime set catches unsupported names, but it cannot prove that a supported name carries the correct payload semantics. Full schema validation catches more mistakes and costs more CPU, maintenance, and rollout coordination. The trade-off is deliberate: start with name validation when silence is the incident, then add payload validation where malformed fields can corrupt state. A tiny embedded client may reasonably perform only the name check and leave deeper validation at the ingest boundary.

Accepted appends minus observed dispatches can help inside a tightly defined synthetic test window. It is not a universal loss metric. Reconnects, filters, and intentionally offline clients change the populations. Scope makes the number honest.

Deploy contract changes in an order that preserves overlap. Consumers must understand a new name before producers emit it. If a rename is unavoidable, define a compatibility window and test both names during that window; do not leave an alias forever. Then remove the old name from the runtime set so validation catches stale producers.

The UI should distinguish connected, no accepted status yet from reconnecting and backfill in progress. Those are operational states. A green connection indicator beside stale device data points the operator toward the wrong subsystem.

What I would change at scale

The three-item tuple works while one team owns producer and dashboard. Across independently deployed teams, I would generate language-specific types and runtime validators from a versioned schema, then add compatibility checks in CI. The price is more machinery: evolution rules, generated artifacts, and rollout discipline. Pay it only when independent releases make handwritten synchronization demonstrably risky.

Retain replay data according to the dashboard's recovery objective. The required window comes from the longest supported disconnection plus operational margin. Storage volume, replay latency, and incident recovery pull in different directions, so benchmark with the actual event size and device update rate. Invented throughput numbers are worse than none.

At scale, test forced disconnects and delayed acknowledgments. Measure time-to-first-live-update separately from time-to-catch-up. One tells you whether the current game floor is visible; the other tells you how long historical state remains incomplete. A single latency percentile blends two jobs.

Prove the event name contract before debugging transport recovery. Then prove append, live dispatch, replay, deduplication, and render as separate boundaries. This ordering turns a silent dashboard into a short chain of falsifiable checks.

Further reading

Top comments (0)