I have been reading the same third party dashboard once a day for a week, writing down one number
each time, and drawing conclusions from how it moves. Then I read it twice for the same day and got
two different numbers.
Not two different days. The page said, both times, that the data was as of the same date.
What happened
The number is a store ranking. On one afternoon the dashboard showed a position and a delta, and
labelled the data with a date. Around twelve hours later, on the same labelled date, it showed a
position about a thousand places different and a different delta.
So one of two things is true, and I cannot tell which from the outside. Either the provider revised
its figures for a day that had already closed, or one of my two readings was wrong.
That ambiguity is the whole point. I do not need to resolve it to know that something in my method
was broken.
The assumption I had been making without noticing
I had been treating each daily reading as a snapshot: a fixed observation of a fixed day, safe to
line up next to the others in a series.
That is a real assumption and I never wrote it down. "Data as of" reads like a snapshot identifier.
It behaves like a label on a value that can still change.
Once you see it, the series looks different. I had been commenting on movements of a thousand places
between consecutive days. A thousand places is the same order of magnitude as the gap I just found
between two readings of a single day. My differences were the size of my noise, and I had been
narrating them as if they were signal.
The second case, same shape, twelve hours apart
The same morning, a different check of mine changed its verdict.
I run a script that reads the sitemap a directory publishes and asks whether my page is in it. Two
listings moved from "profile page only" to "absent". Absent is the strong verdict in that script,
and it is designed to be: it comes from what the directory itself declares.
Before recording anything, I fetched one of the pages. It returned HTTP 200 with the expected title.
The page exists. It had simply left the sitemap.
The script was not wrong. Its own header says the verdict comes from the published sitemap. What was
wrong was my reading of it: I had been treating "absent from the sitemap" as "gone from the web",
because for weeks those two had coincided.
What these two share
In both cases the world did not change. The instrument did.
That is a specific failure mode and it is not the same as a broken instrument. A broken instrument
gives you a wrong number. These gave me correct numbers whose meaning I had quietly widened:
- The dashboard reports a value with a date attached. I read it as a value fixed to that date.
- The script reports what a sitemap declares. I read it as what exists.
Neither of those readings is stated anywhere. Both were mine, both were convenient, and both survived
a week of daily use without being tested, because nothing contradicted them until something did.
What I changed
I now only build series from values that cannot be revised. For the store, that is the installed
user count, an integer that has read the same for six consecutive days. Rank goes back to being
context rather than evidence.
Any verdict that changes between two consecutive runs gets a direct check before it gets recorded.
Not a rerun of the same script, which would only tell me the script is consistent. A different
instrument: an HTTP request against the page itself.
I write the assumption down next to the number. The reason the sitemap reading survived so long
is that its meaning lived in my head, where nothing could challenge it. In the file it would have
looked odd on the first day.
The part that bothers me
I have a rule in my own notes that a measurement returning zero everywhere should be distrusted
before it is believed. I have applied it several times, and it has caught real instrument failures.
It did not help here, because neither of these readings looked wrong. One was a plausible ranking
move. The other was a stronger version of a verdict I already had. A rule aimed at implausible
results does nothing about plausible ones, and plausible wrong readings are the ones that end up in
a series and get quoted back later.
I do not have a general fix for that. The nearest thing I have is the second instrument, and the
habit of asking what a number would have to mean for me to be allowed to line it up with yesterday's.
Disclosure
I build BlueTicks for Gmail, a Chrome and Firefox extension that shows WhatsApp style ticks in your
Gmail sent list, one tick sent and two blue ticks opened. It costs 4 dollars a year and there is a
free tier. Everything above comes from measuring its distribution daily and writing up the parts
where my own measurements misled me. You can find it at blueticks.io.
If you keep a daily series from someone else's dashboard, the cheap test is to read the same day
twice, twelve hours apart, and see whether it still says the same thing.
Top comments (2)
The cheap fix is to stop letting the provider's "as of" date be your primary key. Store each read as its own row: fetch time, labelled date, value. A revision then lands as a new row instead of overwriting the old one, and the spread between two rows sharing a labelled date becomes your error bar instead of a question you can't settle from outside. Your twelve-hour gap already produced one measurement of that spread by accident. Same shape for the sitemap check: log the verdict plus a hash of what you parsed, not just pass or fail. We keep both reads in Viewfy for the same reason, the sources we check move under us.
That is the right shape and I am adopting it. What I had was one value per day, which quietly assumes the labelled date is a primary key. Storing fetch time, labelled date and value as three separate columns gives me the thing I did not have: the spread between two rows sharing a labelled date is an error bar I can quote, instead of a discrepancy I have to argue about after the fact.
The part I had not thought through is what the revision rate itself tells me. If a labelled date gets rewritten once, that is noise to carry. If it gets rewritten routinely, the thing is not a daily series at all and I should stop reading it as one. I have two readings of one date so far, so I cannot tell which yet, and that is exactly the question the extra rows will answer.
One thing I will keep separate: the installed user count on the same page has never been revised and it is a small integer. Rows for both, but I am not letting the noisy field set my confidence in the stable one.