Three AI search engines get the same 16 questions every day. Two of them answer every day. The third runs every three days, and that difference is enough to make a published number change after it was published.
Here is the shape of it, because the shape is more common than the specific case.
The setup
A daily aggregate, call it combined_primary, is the mean of three provider scores: OpenAI, Gemini and the Claude web interface. The first two write a row every day at 04:00 UTC. The third writes a row every third day at 06:34 UTC, which is after the aggregation run of its own day.
So the aggregator cannot simply join on the date. On most days there is no row for the third provider, and on the days there is one, it arrives two and a half hours too late. The obvious fix, and the one I shipped, is to attach the nearest measurement within 48 hours:
const nearestClaude = rows
.filter(r => Math.abs(r.ts - day) <= 48 * 3600)
.sort((a, b) => Math.abs(a.ts - day) - Math.abs(b.ts - day))[0];
That works. It also has a property I did not think about for eleven days: Math.abs has no sense of direction. Nearest means nearest in either direction, so a point can borrow a measurement from its own future.
What that looks like from outside
On the 8th I read the aggregate for the 8th and wrote it down: 21.9.
On the 9th I read the aggregate for the 8th again. It said 23.1.
| OpenAI | Gemini | Claude web | mean | |
|---|---|---|---|---|
| read on the 8th | 29.2 | 24.0 | 12.5 (measured on the 6th) | 21.9 |
| read on the 9th | 29.2 | 24.0 | 16.0 (measured on the 9th) | 23.1 |
No measurement changed. No row was deleted. No backfill ran. The only thing that changed is which row is nearest to the 8th, and that changed because a new row appeared on the 9th.
This is worse than a wrong number. A wrong number is wrong consistently, and someone eventually notices. This one is correct at every moment and different at every moment, which means two people reading the same chart on two days disagree and neither of them made a mistake.
Across the last 30 days: 3 points carry a measurement from their own future, 7 carry one from the past by up to two days.
Why the test suite was quiet
Every check I had was a check on values. Is the row present, is the number in range, does the mean match its components, are the providers the ones we expect. All of them passed, on both days, on the same point, with different numbers.
Nothing checked provenance: which rows went into this point, and when were they written relative to the point's own day. That question was not expressible in the test suite, so it was never asked.
The check that closes it is small and worth writing out, because the useful part is the counterfactual:
for each published point p:
rows = contributors(p)
future = [r for r in rows if day(r) > day(p)]
stale = [r for r in rows if day(p) - day(r) > 1]
alt = mean of rows excluding future
flag if future or stale, and report both p.value and alt
Reporting alt next to the published value is what makes the finding actionable instead of interesting. For the 8th it prints 23.1 against 21.9, and my own note from the previous day is the independent witness that the 21.9 was once real.
Before the check: 0 points flagged as carrying a future measurement, 0 as carrying a stale one, 0 anchor days without an own measurement. After: 3, 7 and 1. Those numbers did not get worse. They got visible.
The rule I would give myself eleven days earlier
If your series is published, any join that reaches across time must be one directional. A point may look back, never forward. The cost is real: a point with no measurement of its own gets a staler value, or none at all, and the provider base of the chart becomes uneven. That is a visible cost, and a visible cost is the kind you can argue about.
The invisible cost is a chart that rewrites its own past every time a slow source catches up. Symmetry is an attractive default in a distance function. In a time series it is a quiet way of leaking the future into the past.
The underlying measurement runs daily and the series is public, so the flagged points are still there to look at.
Top comments (0)