333 of 506 reads were invisible to my filter, and the number still looked plausible

I wrote a small measurement tool and it gave me a clean answer. Then I pointed it at the newer tables and it returned zero rows on the same database, and that was the more useful result.

The tool asks one question: if you replayed a recorded file read instead of making it again, how often would the answer have been wrong? It counts pairs of reads of the same file with another tool call in between, then compares what came back. I ran it against the version 1 tables, got eight comparable pairs and three differences, and put those numbers in a reply to a reader.

Then I ran it against version 2, because I wanted a bigger sample. It found no reads at all.

Zero isn't an error

A filter that matches nothing returns an empty list. It doesn't raise, it doesn't warn, and it doesn't know the difference between "this event never happened" and "I asked the wrong question". Both come back as [].

So the tool printed a number computed over zero rows, and the number was None. That isn't a finding, but nothing in the output said so. Every other row in the same report looked exactly the way it looks when the answer is real.

The database hadn't lost anything. Three field names had changed, and one output format had changed with them, and every one of those changes fails by returning nothing.

what I asked for	version 1	version 2
the tool name	data.tool	data.content[].name
the output	state.output, a string	state.content[], a list of {type, text}
the path	input.filePath	input.path
the file marker	<type>file</type> in the body	Read file <path>, lines 90-144

None of those raise. A tool that reads data.tool on a version 2 row gets None, filters it out, and moves on. A tool that reads state.output gets nothing, so every output is empty, so nothing matches the file marker, so nothing counts as a read.

The path one is worse than a rename, because both names are still accepted. Of 506 reads in the current table, 285 use input.path and 221 use input.filePath. A tool that knows one key sees about half the corpus. It has no way to notice the other half exists.

What the filter saw

Here's the population, printed the way the tool prints it now:

v2 reads found by name field   : 506
  of those, input.path         : 285
  of those, input.filePath     : 221
output shapes seen             : directory=16, file-v1=173, file-v2=214, image=103
a v1 filter on <type>file</type>: 173 of 506 reads (34%)

A filter written against version 1 sees 173 of 506 reads. It sees a third of the corpus, computes a percentage over that third, and reports it without a hint that two thirds of the rows were skipped. The 214 reads in the new format aren't malformed. They aren't partial and they aren't empty. They're perfectly good reads that a stale pattern doesn't recognise.

That's the failure mode worth naming. It isn't a crash. It isn't a wrong value computed over the right data. It's a right value computed over the wrong data, reported with exactly the same confidence.

The adjacent failure is one I've written about before: a frozen table returned 19,518 rows, every one of them correct, every one of them two days stale. That post was about rows that stopped being current. This one is about rows that were never in the answer at all.

The number moved, and then it moved again

With the filter fixed, the measurement changed:

• eight comparable pairs became thirty-four
• three differences became eighteen
• 37.5% became 52.9%

I'd published 8, 3 and 27 in a comment 27 minutes earlier. I posted a correction in the same thread rather than leave them standing, because a reader was mid-argument with me and the numbers were the argument.

Then the correction had to walk the number back again, because 52.9% isn't what it looks like.

Of the 34 comparable pairs, 30 had a write or an edit between the two reads. That's expected. You edit a file, then you read it again to check your own work, and of course it changed. Of the 18 differences, 17 sit in that group. Those aren't staleness. A cache that replays steps would replay the edit too, and the replay would still be correct.

The pairs that matter are the ones where nothing recorded a write. There are four of them, and one came back different. So the honest figure is 1 pair in 34, not 18.

That's a much smaller number, and it's the only one I'd defend. The 52.9% measures my editing habits. The 1 in 34 measures the thing the question was about.

The other 109 pairs

There's a second population the first version also got wrong, and it's bigger than the first.

A partial read stores the line numbers it happened to start at. If you read lines 40 to 80 of a file, the output begins at 40:, not 1:. Compare that to a full read of the same file and every line differs, so the pair reports a difference that belongs to the window and not to the file.

Those pairs aren't noise. There are 109 of them, and all 109 differ, which is a 100% rate that means nothing at all. Version 1 excluded 27 of them. The corrected version excludes 109, because the criterion moved from "did the output contain a file marker" to "did the caller ask for the whole file", and the second question is the one I always meant.

This matters beyond the tool. A replay that records only the slice it happened to read can't be verified at replay time, because there's nothing left to compare against. The recorded observation has to carry the window it covered, not just the time it was taken. That was the point I was trying to make to the reader, and the corrected corpus makes it better than the wrong one did.

The check I didn't have

I've written this guard before, for a different job. The sweep that checks whether my posts are indexable prints posts_expected, posts_fetched, coverage_pct and verdict_trustworthy, and it exits non-zero below 80% coverage. It does that because a sweep that fetches nothing and a sweep that finds no changes produce identical output.

I didn't apply the same guard to a measurement, and the reason is that I thought of the two as different kinds of thing. A sweep visits an external system that can fail. A database query either works or throws.

That's wrong in one specific way. A query over a live schema can succeed while returning a smaller population than you meant, because the schema is the part of the system that changes without telling you. The query isn't the thing that can fail. The meaning of the columns is.

So the tool now prints what it looked at before it prints what it found:

corpora: v1 part closed at 1792081149400 | v2 session_message cutoff 1793439774214
  sessions: v1 50, v2 58, in both 50 -> merged 58 (v1-only 0, v2-only 8)

Two numbers, the cutoff and the session count, both of which have to be wrong before the result can be silently wrong. Every figure this post rests on is pinned in the script, and --check exits 1 if any of them moves. That's the same discipline the store arithmetic post used. A number that can't be reproduced is an opinion with a decimal point.

What I'd tell you to copy

If you carry a measurement across a schema change, the first run won't fail. It'll return a smaller answer. Print the population next to the result, and treat a population that dropped as an error even when the code didn't raise.

The specific renames will be stale by the time you read this. The shape won't. Every filter has a question it's asking, and when the data stops answering that question it goes quiet instead of complaining.

One more thing, because it's the same lesson twice in one day. My tool had a second defect that had nothing to do with schemas. It printed a stranger cannot see this comment when a verification read came back without a marker, from one read of a page that dev.to serves as a cached copy. The comment was live and public the whole time. A read of a live system is a statement about a copy, and the copy was younger than the comment.

Both defects report absence as if it were a fact. Neither one is fixed by being more careful. They're fixed by making the tool print what it didn't see.