We set out to measure something we could not find measured honestly anywhere: how much of the web data sitting in commercial datasets is still true. Not stale in the sense of "this table has not refreshed." Stale in the sense that costs someone money: you bought a row, the row says a page contains a value, and the page does not contain that value anymore. We build independent web data verification, so running the measurement is the natural thing for us to do, and the number would have been the best marketing asset we could ask for.
We ran the study five times in one day. We have no number for you.
The one day is worth explaining before anyone else raises it, because it sounds like haste and it is the opposite. Running a verification pass costs us about $1.10 per thousand rows, so the runs are cheap and fast. What was slow was the rules, which were written first and which is why five runs produced zero publishable numbers instead of one convenient one. Cheap measurement plus strict rules gets you a lot of failures per hour. That is the system working.
This is the report on why, and it is more useful than the number would have been. The rules we wrote before we looked
Before the first run we wrote down and committed the method: which datasets we would pull from, how we would search for them, how the sample would be selected, what would disqualify a run, and what a pilot run is allowed to produce. That last pair is what did the damage.
We pre-registered an honest-N floor. The study targets 25 usable datasets. Below 10, the run does not publish at all, no matter how interesting the result looks. We wrote that rule before we had any results, precisely so we could not decide afterward whether our sample was big enough. A sample size chosen after seeing the data is not a sample size, it is a preference.
We also pre-registered that pilot runs never publish their numbers, and
deterministic sampling, so that which rows got tested was decided by a hash rather than by us. From the scale run onward we recorded parse coverage as a number on every dataset, so rows we could not read were counted rather than quietly dropped.
Every one of those rules has now cost us something. That is what they were for.
Run one: the search was wrong
The first run returned nothing usable. The cause was not the engine and not the data. Our dataset search matched on names rather than descriptions, so the protocol was looking through a keyhole and reporting an empty room. We amended the search protocol and recorded the amendment. We did not rerun and quietly call it run one.
Run two: our own spot check was measuring the wrong thing
The second run came back with 28 of 28 rows unverifiable.
The interesting part is why we did not catch it earlier. Our pre-run spot check tested whether a sample row's URL was reachable. Reachable is not the same as addressable by the engine, and the difference is the entire study. A page can answer a request and still be something our verification cannot form an anchored question about.
We amended the spot check to require rows that are engine-addressable through the actual verification path, not rows that merely return HTTP 200 to a browser.
Run three: a number we are not going to show you
The third run was the pilot, and it produced an actual figure.
Pilot numbers never publish. We wrote that down before the run, so the figure is recorded in our results file and excluded from publication.
We are aware of how that reads in an article about honesty, so let us be exact. We are not withholding it because it embarrasses us. We are withholding it because the rule that governs it was written before we knew what it would say, and a rule you suspend the moment it becomes inconvenient was never a rule.
Publishing the number here, framed as "the one we could not publish," would be publishing it. If a later run clears the bar, you get that number with its full method attached. If none ever does, you never get one, and that will also be true. The scale run, and the finding actually worth your time
The fourth attempt was the big one: 17 datasets, 1,694 rows. It produced no publishable number either. It produced something better.
The results came back bimodal, which tripped a flag we had pre-registered for exactly this shape. Bimodal means two populations are hiding inside one average, and an average across two populations is a lie with a decimal point in it. The flag forced us to characterize before reporting.
Characterizing found three defects, all three in our instrument rather than in the world.
One: we were reading the wrong column. Some datasets carry several URL
fields, and the one we treated as the product page pointed at image assets. We were checking whether a price appeared on a JPEG.
Two: claimed values carried prefixes that were never on the page. A row
whose value arrives with a vendor-added label attached fails verification for a reason that has nothing to do with data quality. The value was right. The string was not.
Three, and this is the one to take away: bot walls look exactly like stale
data. A meaningful share of our failures were sites returning a challenge or interstitial page at HTTP 200. The request succeeds. The status code says everything is fine. The page contains no product and no value. To a verification pass that is indistinguishable from a page whose value has changed, and it scores as a confident failure.
The implication is worth stating plainly and carefully. Any measurement of web data decay that does not screen for challenge pages served at HTTP 200 will overstate decay, and will not be able to tell you by how much. We do not know how many published figures carry that defect. We know ours did, and we only caught it because a flag we had written in advance refused to let us average across it.
We now keep a frozen list of seventeen challenge-page entries, one of which is a paired rule. It was committed before the next run started, and any addition made mid-study is logged as an amendment rather than applied quietly, because a detection list you expand while looking at results is a way of choosing your answer.
The control, and why we threw it out
While characterizing, we ran a control against a dataset with clean product URLs. It passed at 95 percent, which told us the instrument behaves when the inputs are clean.
Then we disqualified it. The control scraped a website that exists specifically for people to practice scraping on. It is static by design. It never changes, which is exactly why nothing on it had decayed, and exactly why it can tell you nothing about the world. The sandbox-exclusion rule we wrote afterward rejects it, so we applied that rule to our own most flattering result.
Our best-looking number came from a training site. We are telling you that
instead of letting you find it. The rerun, and the fifth wall
With the defects fixed we reran. The corrected harness screens challenge pages before they reach the engine, prefers page URLs over asset URLs, tests both raw and sanitized claim strings, excludes sandboxes, and caps how many datasets any single publisher can contribute so one source cannot carry the study.
Screening that aggressively removed a great deal. The run landed at 5 usable datasets, below our own floor of 10. So it does not publish. That is the fifth time this study has produced no number and the third time it produced one we walled.
What we actually know now
We know our search protocol was wrong and we fixed it. We know our spot check was testing the wrong property and we fixed it. We know at least three ways to accidentally manufacture a frightening decay statistic, because we manufactured all three. We know the same mistake wore two different disguises in one day: a vantage that could not reach a page, and a vantage that reached a wall and mistook it for an answer. Both times we were describing our own position and calling it the world.
We do not know how much commercial web data is stale. We are further from
claiming to know it than we were this morning, and that is the correct direction to be moving.
Why publish this
Because the alternative was available and we did not take it.
We could have published the pilot figure with a caveat paragraph. We could have published the pre-fix scale run, which produced a much more alarming number than the corrected one, and no reader could have caught the image-column defect from outside. We could have kept the 95 percent and left out where it came from. All three would have drawn more attention than this article will, and all three would have been the kind of vendor study that this industry has correctly learned to discount.
We sell independent verification. The only thing we are asking anyone to buy is that we do not round toward the answer we want. A company like that publishes its walls, or its numbers mean nothing when it finally has some.
Check us, and the limit of checking us
The pre-registration, every amendment with its date, the results, the decision log and the frozen challenge-page list are published alongside this article, with dataset identities replaced by opaque labels. The identities are withheld because we will not name someone's dataset in a study about failure without their consent, and we publish a commitment to the private mapping so that if anyone does consent later, the reveal is provably the same mapping and not a convenient one.
Here is the limit, since this is an article about limits. Our record shows the rules were written before the runs, but that record is ours. Timestamping it now would prove it existed now, not that it existed first, and we are not going to dress up a proof we do not have. From the next edition onward the pre-registration is timestamped by a third party at commit time, so the ordering becomes something you can check instead of something we assert.
We will run this study again. If it clears the bar, you get the number. If it does not, you get this instead.
Rick C.
Fieldmode LLC
Top comments (0)