Every classification system has an error rate. This is uncontroversial for spam filters and recommendation engines, where the cost of a mistake is a misplaced email or a bad suggestion. It becomes a design problem when the classification is a public claim about whether something is true.
I want to work through why verdict-style tools degrade as they scale, and why the degradation is structural rather than a matter of model quality.
The asymmetry nobody budgets for
Suppose a system labels claims as true or false with 95% accuracy. That sounds good. At a thousand claims a day, it produces fifty wrong labels a day.
The problem is not the fifty. It is that the fifty are not randomly distributed in their consequences.
A false negative — a misleading claim labelled true — is quiet. Nobody notices, nothing happens, the error stays in the system. A false positive — a legitimate claim labelled false — is loud. The person who made the claim notices immediately, and they have every incentive to publicise the error.
So the visible error rate is not 5%. It is whatever fraction of the 5% is both wrong and consequential, and that subset is heavily biased toward cases where someone is motivated to argue.
At small scale this is manageable. At scale, the loud errors accumulate into a reputation, and the reputation determines whether anyone accepts the next verdict.
Why accuracy does not rescue you
The intuitive fix is to improve accuracy. Ship a better model, reduce the error rate, reduce the consequences.
This helps less than it should, for two reasons.
First, the remaining errors concentrate in exactly the cases that matter. Easy claims get easier as models improve; contested claims stay contested, because they are contested for reasons that are not primarily about information. A system at 99% accuracy is still wrong on the claims people argue about, because those are the hard ones.
Second, higher accuracy raises the stakes of each error. A system that is right 95% of the time is treated as a heuristic. A system that is right 99% of the time is treated as an authority — and an authority that is wrong is a scandal, not a rounding error.
You can improve the model. You cannot improve your way out of the structural position of being an arbiter.
The trust prerequisite
There is a deeper problem, which I have written about before but which is worth restating in this context.
To accept a verdict, you must already trust the source of the verdict. A "false" label is only information if you believe the labeller.
For a reader who is already convinced that a domain is biased — politics, health, anything contested — the label is not evidence. It is a claim by an entity on the other side. The tool has no purchase on the audience it was designed for, and the audience it does work on did not need it.
This is why verdict tools tend to serve as confirmation infrastructure. They get used by people who already agreed, to settle arguments with people who were never going to accept the settlement.
What changes if you drop the verdict
The alternative is to describe structure instead of assigning truth values.
Rather than "this article is misleading", the output is: these sentences are verifiable claims, these are opinions stated in factual form, these claims have no attributed source, this comparison lacks a baseline.
Three things change immediately.
The error surface shrinks. "This sentence contains no attribution" is checkable by the reader in about two seconds. A wrong classification is visible and correctable, not a contested judgement. You are making small claims that can be individually wrong without discrediting the whole.
Disagreement becomes partial. If you disagree with a verdict, you reject the tool. If you disagree with one classification out of forty, you note it and keep reading. Partial disagreement is compatible with continued use; total disagreement is not.
The trust requirement inverts. Structural claims can be verified without trusting the system, because the evidence is the text itself. The reader does not need to believe you about whether a source was cited — they can look.
The cost of this approach
I am not presenting this as free.
Structural output is more work to read. A verdict is one bit; a structural analysis is a document. Most users want the bit, and telling them the bit is unavailable is a real product cost.
It also declines to answer the question people actually asked. Someone who wants to know whether an article is trustworthy is not fully served by "here are eleven unattributed claims" — that is an input to their judgement, not a substitute for it. Some users experience this as evasion.
And structural analysis has its own failure modes. It is sensitive to input length: analyse three paragraphs and you get an assessment of three paragraphs, which may differ substantially from the full text. Fragments produce confident readings of things that were not there. That is a property of reading structure, not a defect to be fixed, and it needs to be stated rather than hidden.
Where this leaves error handling
The design question I keep returning to: what does your system do when it is wrong, and who finds out?
Verdict systems fail silently in one direction and catastrophically in the other. Structural systems fail visibly and locally — a wrong classification sits next to the sentence it describes, where the reader can see it.
That is not better accuracy. It is a different distribution of the same errors, arranged so that the ones that survive are the ones with the smallest consequences.
For anyone building classification tooling in contested domains, I would argue that the distribution matters more than the rate. A system that is wrong 10% of the time in ways that are immediately checkable is more usable than one that is wrong 2% of the time in ways that require trusting it.
We publish the individual analyses this approach produces on faktoskop.pl, including the ones where the assessment is unflattering to the method — which is the other half of the argument, but that is a separate post.
Question for the community
For those who have shipped classification systems where errors are publicly visible: how do you think about the difference between error rate and error distribution? Most of the literature I can find optimises the former and is quiet about the latter.
Top comments (0)