Why Yesterday's Event Report Changed Today: A Snapshot Test for Late Arrivals

A dashboard can show a higher count for yesterday without either query being wrong. An event may have happened yesterday, arrived after the first report ran, and been included in a later run. To investigate, keep event time, arrival time, and the report's as-of cutoff separate.

This AI-generated engineering note is published from the SensorFlow project account. Its synthetic code was executed locally, and the linked documentation was checked. The example is synthetic JavaScript. It is not an SDK, pipeline, or production reliability test, and it does not claim that any named analytics product loses events.

A report needs two boundaries

The event-time window answers which day an action belongs to. The as-of cutoff answers which records were available to that particular report run. For this example, both are explicit UTC instants. A record occurring at 23:59Z but arriving at 00:10Z belongs to the earlier day, yet a snapshot made at 00:05Z cannot include it.

Apache Beam's time model distinguishes event time from processing time and discusses watermarks as estimates of completeness. The idea is useful even when no Beam runner is involved: a clock passing midnight does not prove that every event for the day has arrived.

Run a small snapshot test

Save this as snapshot.mjs, then run node snapshot.mjs. The rows are made-up records, not captured customer data.

import assert from 'node:assert/strict';

const rows = [
  { id: 'a', occurred: '2026-10-14T10:00:00Z', arrived: '2026-10-14T10:01:00Z' },
  { id: 'b', occurred: '2026-10-14T23:59:00Z', arrived: '2026-10-15T00:10:00Z' },
  { id: 'c', occurred: '2026-10-15T00:02:00Z', arrived: '2026-10-15T00:03:00Z' }
];

function instant(value) {
  assert.match(value, /Z$/);
  const time = Date.parse(value);
  assert.ok(Number.isFinite(time));
  return time;
}

function dailyCount(input, asOf) {
  const cutoff = instant(asOf);
  const start = instant('2026-10-14T00:00:00Z');
  const end = instant('2026-10-15T00:00:00Z');
  const seen = new Map();
  for (const event of input) {
    const arrived = instant(event.arrived);
    const occurred = instant(event.occurred);
    if (arrived >= cutoff || occurred < start || occurred >= end) continue;
    if (seen.has(event.id)) assert.equal(seen.get(event.id), occurred);
    else seen.set(event.id, occurred);
  }
  return seen.size;
}

assert.equal(dailyCount(rows, '2026-10-15T00:05:00Z'), 1);
assert.equal(dailyCount(rows, '2026-10-15T00:11:00Z'), 2);
assert.equal(dailyCount([...rows, rows[0]], '2026-10-15T00:11:00Z'), 2);
assert.equal(dailyCount(rows, '2026-10-15T00:10:00Z'), 1);
assert.equal(dailyCount([...rows].reverse(), '2026-10-15T00:11:00Z'), 2);
assert.throws(() => dailyCount([...rows, { ...rows[0], occurred: '2026-10-14T11:00:00Z' }], '2026-10-15T00:11:00Z'));
console.log('PASS: six synthetic snapshot checks');

The first snapshot counts only a. The later one counts a and late arrival b. Event c belongs to the next UTC day. A repeated a does not increase the logical count, while a conflicting occurrence time for the same ID is rejected by this teaching rule. The cutoff uses a half-open interval: a row arriving at the cutoff is excluded.

This example assumes the ID is stable and unique in its intended scope. A real system needs a scoped event key, conflict handling, versioned event definitions, and a defined arrival timestamp. It also needs the business's chosen calendar timezone. JavaScript Date stores instants; local and UTC calendar interpretations differ, while Date.parse can be unreliable for nonstandard strings. The explicit Z strings here avoid machine-local date parsing. The test uses Node's strict assertions.

Explain revisions instead of hiding them

A useful daily report records its date window, timezone, as-of cutoff, and whether it is provisional. When a count changes, compare the two snapshots by event ID: which rows arrived later, which were duplicates, and which changed definition or were rejected? That makes “late data” a testable explanation rather than a guess.

The six checks above verify only the synthetic rule. They do not establish a universal wait period, an exactly-once delivery guarantee, or end-to-end event completeness. A real acceptance test must compare application outcomes, capture attempts, receiver responses, stored rows, and report filters under one declared contract.