My dashboard had a column called "today". Today wasn't finished.
I build devdigest, a daily tech digest. Every change that changes behaviour gets reviewed against real data five days later, and I write the numbers down before I let myself have an opinion about them. It is the only thing standing between me and believing whatever I was hoping for.
This week's series, the share of each digest landing in the four categories a fix I had just shipped was meant to feed:
69.2, 67.6, 60.0, 53.0
Four days, straight down. I had the write-up half composed in my head.
Then I noticed the last number was wrong.
The bug
Subscribers choose when their digest arrives. The first goes out just after five in the morning, the last at midday. My measurement treated the current day as a finished one, so reading it in the morning meant measuring a day that had not happened yet.
The real figure was 53.9, not 53.0. Nine tenths of a point. Not exciting. Here is why it is worth writing down.
Why a small error mattered
The error is directional, not random. The late digests skew one way, so an early reading is always wrong in the same direction. And it is always wrong on the newest point, which is exactly where you look when you are deciding whether something is trending.
A random error adds noise you can see. A directional error on the most recent point manufactures a slope.
The part I got wrong twice
Day five came in at 61.9 and did not continue the decline.
My first instinct was to write that up as the decline never having been real. That instinct is wrong, and it is wrong in the same way as the one it replaced. Four points looked like a trend. Five points look like a recovery. Neither is something five points can tell you, and day six could resume the fall.
What I can say is narrower:
| Observed four-day drop | 15.3 points |
| Largest four-day drop in the 22 days before it | 8.7 points |
So something real happened, and waving it away as noise would have been the mirror image of the mistake I had just caught myself making.
The measurement error and the reading error turn out to be the same error wearing different clothes: wanting a small number of days to settle a question they cannot settle.
So I am not calling it. There is a date in my notes to read the same series over eleven days instead of five, and I will post what it says.
The fix
The guard is small. Metrics now stop at the last day that is provably closed:
def _last_complete_day() -> str:
"""The most recent date whose rows are final: yesterday, UTC."""
return (date.fromisoformat(today_str()) - timedelta(days=1)).isoformat()
Yesterday is provably safe here rather than merely conservative, and I checked before choosing it: every row in the table was created and last updated within its own date, across the whole retention window, so once the date has passed nothing further is written to it.
A partial day does not look like missing data. It looks like a data point, and you will believe it.
If anything you rely on has a "today" column, go and find out what time today finishes.
Top comments (0)