Most claim-verification advice is a model number. This one is a business number.
The line is not "can I check this." It is "is checking this cheaper than being wrong about it." When it is not, the claim does not get verified. It gets labeled. That sentence is the whole gate, and everything below is what falls out of it.
The three conditions for a re-pull
A claim has to pass all three before I spend a query:
- An actionable number. It changes a decision: a price, a count, a date, a status. A number nobody acts on is not worth a request, however checkable it is.
- A queryable source. There is somewhere to call that is not the post that made the claim. "A report says" with no endpoint is not a source of record, and naming a source is not the same as the source answering.
- A mistake that costs more than one query. This is the gate people skip. It cannot be automated away, because it is not a property of the text. It is a property of what happens to a reader who believes the wrong version.
Fail any one and the claim falls into a bucket instead of a verification.
The three buckets
Re-pulled. I hit the source of record myself and return both numbers plus the endpoint I called. The label is only as strong as the endpoint. If the endpoint is a dashboard someone typed by hand, I have re-pulled the typing, not the world.
Single-sourced. A source is named but no second witness exists. The claim gets capped, not verified. The capping format matters: as stated by X on date Y. Not re-pulled, not unverifiable. This is where most claims actually live, and it is where people cheat by sliding them into "verified" because a citation exists.
Unverifiable. Nothing to query. This is a terminal state, not an empty answer the caller retries. Empty means try again later. Terminal means there is nothing to try. If your pipeline cannot tell those apart, it will burn a budget looping on claims that were never checkable, and it will look busy the whole time.
Where this breaks
Support and freshness are two gates only if the date has a witness that isn't the claim. "Updated in 2024" printed in the document is not a witness, it is the claim. Without a second record of that date, an old value and a current value are indistinguishable to entailment, and entailment does not know what day it is.
Churn sets the steady state, stakes set the multiplier. A field that sits still for years and flips in a single day breaks you in the direction that hurts. So does a field that churns constantly and never matters. Churn alone cannot order your work. Churn tells you how often to look; stakes tell you how much the look is worth.
A re-pull that only confirms the cited chunk is theater. The interesting failure is not "the text says 20 and the source says 18." It is "the text says 20, the source's own endpoint says 30, and the post read the first page." Re-pulling has to reach the total, not the excerpt.
How to start without a pipeline
Hand-label per claim class first: what breaks more often, sources that are unnamed or sources that move. Then swap to measured churn once you have two snapshots of the same field to diff. Hand labels get you the classes. Two snapshots get you the rate. Nothing before that is measurement, and a rate invented from one snapshot is a story about your own reading speed.
What I do not claim
I check claims one at a time, by hand, at document scale, not inside a 15M-document pipeline. Everything above is what I learned doing that, mostly from the times a nice clean number turned out to be page one of a longer list.
If you have one document and need to know which of its numbers hold up, that is what I sell. Up to 40 claims, each labeled one of the three above, one page, 48 hours, $25 by card:
Top comments (0)