I have been reading the same third party dashboard once a day for a week, writing down one number
each time, and drawing conclusions from how it moves. Then I read it twice for the same day and got
two different numbers.
Not two different days. The page said, both times, that the data was as of the same date.
What happened
The number is a store ranking. On one afternoon the dashboard showed a position and a delta, and
labelled the data with a date. Around twelve hours later, on the same labelled date, it showed a
position about a thousand places different and a different delta.
So one of two things is true, and I cannot tell which from the outside. Either the provider revised
its figures for a day that had already closed, or one of my two readings was wrong.
That ambiguity is the whole point. I do not need to resolve it to know that something in my method
was broken.
The assumption I had been making without noticing
I had been treating each daily reading as a snapshot: a fixed observation of a fixed day, safe to
line up next to the others in a series.
That is a real assumption and I never wrote it down. "Data as of" reads like a snapshot identifier.
It behaves like a label on a value that can still change.
Once you see it, the series looks different. I had been commenting on movements of a thousand places
between consecutive days. A thousand places is the same order of magnitude as the gap I just found
between two readings of a single day. My differences were the size of my noise, and I had been
narrating them as if they were signal.
The second case, same shape, twelve hours apart
The same morning, a different check of mine changed its verdict.
I run a script that reads the sitemap a directory publishes and asks whether my page is in it. Two
listings moved from "profile page only" to "absent". Absent is the strong verdict in that script,
and it is designed to be: it comes from what the directory itself declares.
Before recording anything, I fetched one of the pages. It returned HTTP 200 with the expected title.
The page exists. It had simply left the sitemap.
The script was not wrong. Its own header says the verdict comes from the published sitemap. What was
wrong was my reading of it: I had been treating "absent from the sitemap" as "gone from the web",
because for weeks those two had coincided.
What these two share
In both cases the world did not change. The instrument did.
That is a specific failure mode and it is not the same as a broken instrument. A broken instrument
gives you a wrong number. These gave me correct numbers whose meaning I had quietly widened:
- The dashboard reports a value with a date attached. I read it as a value fixed to that date.
- The script reports what a sitemap declares. I read it as what exists.
Neither of those readings is stated anywhere. Both were mine, both were convenient, and both survived
a week of daily use without being tested, because nothing contradicted them until something did.
What I changed
I now only build series from values that cannot be revised. For the store, that is the installed
user count, an integer that has read the same for six consecutive days. Rank goes back to being
context rather than evidence.
Any verdict that changes between two consecutive runs gets a direct check before it gets recorded.
Not a rerun of the same script, which would only tell me the script is consistent. A different
instrument: an HTTP request against the page itself.
I write the assumption down next to the number. The reason the sitemap reading survived so long
is that its meaning lived in my head, where nothing could challenge it. In the file it would have
looked odd on the first day.
The part that bothers me
I have a rule in my own notes that a measurement returning zero everywhere should be distrusted
before it is believed. I have applied it several times, and it has caught real instrument failures.
It did not help here, because neither of these readings looked wrong. One was a plausible ranking
move. The other was a stronger version of a verdict I already had. A rule aimed at implausible
results does nothing about plausible ones, and plausible wrong readings are the ones that end up in
a series and get quoted back later.
I do not have a general fix for that. The nearest thing I have is the second instrument, and the
habit of asking what a number would have to mean for me to be allowed to line it up with yesterday's.
Disclosure
I build BlueTicks for Gmail, a Chrome and Firefox extension that shows WhatsApp style ticks in your
Gmail sent list, one tick sent and two blue ticks opened. It costs 4 dollars a year and there is a
free tier. Everything above comes from measuring its distribution daily and writing up the parts
where my own measurements misled me. You can find it at blueticks.io.
If you keep a daily series from someone else's dashboard, the cheap test is to read the same day
twice, twelve hours apart, and see whether it still says the same thing.
Top comments (4)
Follow up to the thread above, since the suggestion in it turned into a number.
Rows now carry fetch time, source, field, the labelled date, and the value. Five readings share the label 2026-08-09, holding two distinct values, and the spread between them is 1063 places. That is the error bar for this field on this source, and I had no way to state it before the rows existed.
What it condemns is my own commentary. I had been reading day to day movements of around a thousand places as though they meant something. A thousand is smaller than the error bar, so those were noise I was narrating. The rank has been stable since the first of the two values, four identical readings, so the revision happened once rather than continuously.
Two extraction bugs surfaced only because I started writing to a file instead of printing. My patterns looked for a number before the label, and the page actually reads USERS 40, with the number after. The first version wrote the bare label into the store, which is how the rule that a value containing no digit is not a measurement got added.
The part I would not have reached on my own is the reframing: the labelled date is not the key, the fetch time is. Once that holds, a revision stops being an anomaly to argue about and becomes a row.
The cheap fix is to stop letting the provider's "as of" date be your primary key. Store each read as its own row: fetch time, labelled date, value. A revision then lands as a new row instead of overwriting the old one, and the spread between two rows sharing a labelled date becomes your error bar instead of a question you can't settle from outside. Your twelve-hour gap already produced one measurement of that spread by accident. Same shape for the sitemap check: log the verdict plus a hash of what you parsed, not just pass or fail. We keep both reads in Viewfy for the same reason, the sources we check move under us.
That is the right shape and I am adopting it. What I had was one value per day, which quietly assumes the labelled date is a primary key. Storing fetch time, labelled date and value as three separate columns gives me the thing I did not have: the spread between two rows sharing a labelled date is an error bar I can quote, instead of a discrepancy I have to argue about after the fact.
The part I had not thought through is what the revision rate itself tells me. If a labelled date gets rewritten once, that is noise to carry. If it gets rewritten routinely, the thing is not a daily series at all and I should stop reading it as one. I have two readings of one date so far, so I cannot tell which yet, and that is exactly the question the extra rows will answer.
One thing I will keep separate: the installed user count on the same page has never been revised and it is a small integer. Rows for both, but I am not letting the noisy field set my confidence in the stable one.
Reproduced today, which I did not expect this fast.
Yesterday's spread, between two readings sharing the labelled date 2026-08-09, was 1063 places. Today four readings share the labelled date 2026-08-10 and hold two values, and the spread is 1070. Two labelled days, measured independently, same order of magnitude.
So the noise on this field is not an artefact of one bad day. It is roughly a thousand and seventy places, and it reproduces. Anything smaller than that, including every day to day movement I have commented on since the sixth, says nothing at all.
The part I want to flag back to you is that the rows are what made the second one visible. Today's later reading would have overwritten the earlier one under my old scheme, and the disagreement would simply not have existed as data. That is twice in eight hours that the same design change paid, which is a better return than I get from most things I build.
One field on that page has not been revised once: the installed user count, four readings today, same integer. I am treating that as the only quantity on the page I can put in a series.