Every tool we buy now gives us a confident answer. The AI assistant that says your migration is safe. The scanner that says the dependency is clean. The model that scores a stock 9 out of 10. The output is fluent, fast, and almost impossible to argue with.
I spent last week comparing tools in one such category, and the thing I expected to be the deciding factor — accuracy — turned out to be unusable as a criterion. None of them can prove accuracy to you before you rely on them, and all of them claim it.
What actually separated them was something much easier to check: does the tool tell you what would make it wrong?
That question turns out to generalise well beyond the category I was looking at, so it seems worth writing down.
The problem with a confident answer
A score is unfalsifiable in practice.
If a tool rates something 9/10 and it fails, nothing about the rating was wrong in a way you can point at. The score was a probability. Probabilities are sometimes disappointed. The model re-scores tomorrow and the previous number quietly disappears.
This isn't dishonest. It's a structural property of compressing an argument into a number. But it has a consequence that took me a while to see clearly:
A view that cannot be wrong cannot teach you anything.
If you acted on a 9 and it didn't work out, you've learned nothing about why. You can't tell whether your reasoning was bad or the world changed. You have no basis for updating anything, so the next decision is exactly as blind as the last one.
The same applies to a green checkmark on a security scan, or an AI assistant saying "this refactor is safe."
Four questions that actually discriminate
These are cheap to ask and hard to fake.
1. What would have to be true for this to be wrong?
Ask the tool, or ask the vendor. You're listening for a specific, checkable condition — "this holds unless the input cost moves more than 15%" — versus a restatement of the conclusion.
Most tools restate the conclusion. That's the signal.
2. Can I see what it said last month?
Most tools present a current state. Yesterday's answer is gone.
This matters more than it sounds, because it's the only mechanism by which a tool can ever be held accountable. If the record silently rewrites itself, nobody — including the vendor — can tell whether it's improving.
Try to retrieve what a tool told you a quarter ago. The answer is more revealing than any benchmark.
3. Are the numbers computed, or generated?
In a serious system, the figures are calculated deterministically from source data and the language model writes only the prose around them. In a wrapper, the model produces the numbers too.
That's why wrapper output is occasionally plausible and wrong, which is the worst failure mode available — plausible-and-wrong survives review in a way that obviously-wrong doesn't.
Ask where a specific number came from. You want a source and a date.
4. Does it ever decline to answer?
A tool that produces a confident answer every single time is not more capable than one that sometimes says "the evidence here is mixed." It's just tuned differently, usually because declining is commercially unattractive.
The ability to return "I don't have a view" is a feature. Its absence tells you the tool is optimising for the feeling of an answer.
The worked example
The category I was looking at was AI stock analysis tools, and the pattern was remarkably consistent.
Almost every product in the category outputs a score: a 1–10, a 1–9, a letter grade A–F, a five-dimension visual. They differ mainly in coverage — from a few hundred instruments to well over a hundred thousand — and in how much of the reasoning they expose.
Not one of the tools I surveyed publishes explicit invalidation conditions. Scores update silently. A rating moves from A to B with no moment at which the previous view is marked incorrect.
Several publish backtested performance figures instead. Those numbers may be computed perfectly correctly, and they're still doing something different from what they appear to do: they're vendor-published, largely backtested rather than live, and not independently audited. A performance figure you cannot reproduce is an advertisement.
Disclosure: I work on Synoptiv, which is one of the tools in that category. So treat the next paragraph with the scepticism it deserves.
We went the other way — written analysis instead of a score, with explicit invalidation conditions stating what would make the conclusion wrong, and versioned analyses so a prior view stays on the record instead of being overwritten. We also publish a "no view" outcome when the evidence is genuinely mixed.
I think that's the honest design. But the whole point of this post is that you shouldn't take a vendor's word for which side of a trade-off to be on — so the useful thing I can offer isn't the claim, it's the four questions, which apply to us exactly as much as to anyone else. The full comparison, including the things we're straightforwardly worse at, is here.
Why this generalises
Software is filling up with tools that emit confident judgments: code review bots, vulnerability scanners, observability systems that tell you what caused the incident, agents that decide a change is safe to merge.
All of them face the same design choice. You can present a verdict, which is fast and feels authoritative. Or you can present a verdict plus the conditions under which it fails, which is slower, less impressive in a demo, and the only version anyone can learn from.
When you're evaluating one — or building one — the question isn't "how accurate is it?" You can't answer that in advance and neither can the vendor.
The question is: when this is wrong, will I find out?
I write about how to read financial data and where automated analysis breaks down at synoptiv.com/insights. Nothing here is investment advice.
Top comments (0)