Less technical than my usual posts. Nothing to install, nothing to pay for, and it works the same whether you're using an AI assistant for code, spreadsheets, research or writing.
Here is the shape of the problem I want to describe. See if you recognise it.
You ask your assistant to do something that produces a result. It does. A number, a summary, a status, an answer comes back. It's plausible. You ask why, and you get a clear, confident explanation that makes complete sense of it. So you believe it, you write it down, and you build your next three decisions on top of it.
A week later you find out the whole thing was wrong. Not wrong in an obvious way. Wrong because the tool was measuring something you didn't ask about, or reading a file that was empty, or answering a slightly different question than the one in your head.
Notice what didn't happen there. Nothing was invented. No fake citation, no made-up fact, none of the failures people mean when they say "hallucination". The failure was structural: nobody committed to what the answer should be while it was still possible to be wrong.
The habit
Before your assistant runs anything that produces a result, make it answer two questions.
1. What do you expect the result to be, and why?
2. How would you know if this were broken?
That's the whole technique. Two sentences before the work instead of a paragraph of explanation after it. It works on the free tier. It works on any model. It costs a few seconds.
Why the first question works
This isn't about "thinking harder", and it isn't a motivational trick. It changes the kind of problem your assistant is solving.
Explaining a result you've already seen is an open-ended task. There are a huge number of stories that fit any given outcome, and a language model is exceptionally good at finding one of them. That's not a flaw, it's the thing it's built to do. The problem is that a fluent explanation tells you nothing about whether the result is correct. It reads identical either way.
Predicting a result you haven't seen is a closed task. Everything the assistant believes about your data, your files and your request has to collapse into one statement that can turn out to be wrong. And when the real output disagrees with the prediction, you get a signal that could not have been manufactured after the fact, because the prediction is already sitting there on the screen.
That gap between expected and actual is the entire product. It's cheap to produce and impossible to fake in hindsight.
There's a bonus effect, and it's the reason results feel steadier once you do this. Predicting first pins down the standard of success before the data arrives. Most of the wobble between one run and the next isn't the model changing its mind, it's the standard quietly moving to fit whatever showed up.
Science has a name for this and an entire apparatus around it: pre-registration. Nobody needed to invent it for AI. We just forgot to apply it.
Why the second question matters more
I learned this one the expensive way.
I had a system meant to answer time-sensitive questions: which of these documents was true as of a particular date. It scored badly. I had predicted it would score badly, because the task is genuinely hard. Prediction matched result. I nodded, wrote the number down, and moved on.
The prediction was right and the measurement was worthless.
None of the documents had dates attached to them. Not one, out of four hundred and twenty. The system was being asked a question about time using material that had no time in it. When I fixed that, with exactly the same documents, the score went from 0.19 to 0.98.
I want to be careful with that number, because it's the kind of number people quote badly, including the person who produced it. That is not an improvement. Nothing got smarter. That's what it looks like from the outside when a broken instrument gets repaired, and if I'd published it as a win I'd have been telling a lie with correct arithmetic in it.
Here's the general rule, and it's the part almost everyone skips:
Predicting the answer will not tell you the question was never asked. If the setup is broken, your prediction and your result can be wrong in the same direction, agree with each other perfectly, and confirm nothing.
Which is exactly what the second question is for. "How would I know if this were broken?" forces something different out of the assistant: not a guess at the answer, but a statement about the machinery, one that has to hold true whatever the answer turns out to be.
- Expectation: "the score will be low, this is a hard problem."
- Broken-check: "if the documents don't have dates on them, the question isn't being asked at all."
The first one was right and taught me nothing. The second one would have gone off on day one.
Done is not the same as proved
That second question generalises into something I now say out loud several times a week.
When a task reports success, what you have learned is that a task reported success. You have not learned that a file was written, that the file has anything in it, that the numbers cover the dates you asked for, or that the check you requested was actually performed.
My favourite example is small and stupid. I once wrote a rule-checker to catch a specific kind of mistake in my own work. It ran clean for weeks. It ran clean because of a settings mismatch that meant the rule I cared about was never switched on. The checker was structurally incapable of finding anything. Nothing was broken, nothing errored, everything was green, and the green meant nothing at all.
The first test I wrote to fix that read the settings file to confirm the settings file. It passed. Two separate reviews walked straight past it. The thing that finally caught anything was running the checker against a deliberately bad example and watching it complain.
So: ask for the output, not the status. Show me the rows. Show me the file. Show me it catching one.
Anything that's supposed to warn you, make it warn you once
Same idea, pointed somewhere useful.
A safety net you have never seen catch anything is not a safety net. It's a guess in the shape of one. This applies to far more than code: the alert that's supposed to email you when a number goes out of range, the filter that's supposed to catch the bad rows, the check that's supposed to stop the bad export.
So test it deliberately. Feed it the bad thing on purpose, once, and watch it fire.
I ran that exercise across a system I'd been maintaining for months. It found twelve warnings that could never have gone off under any circumstances. One of them was the reason a cost had been quietly running for about two months before anyone noticed. Every one of those twelve had been read and approved by someone, including me. Reading them was never going to work, because they read correctly. That's precisely what makes them dangerous.
Two things I got wrong before I got them right, and they generalise:
Breaking it on purpose only proves whatever you actually broke. I once "confirmed" that two checks happened in the right order by deleting one of them entirely. All that proved was that a check existed. The ordering, which was the actual thing I cared about, stayed untested.
A check that can't pass is as useless as one that can't fail. I hit five in a row where the thing also refused to go green on a known-good input, and each one had been introduced while fixing the previous one. Identical behaviour on the good case and the bad case means it's measuring nothing, in either direction.
Look it up, don't remember it
One more, cheaper than everything above. When the assistant needs a fact it could go and check, make it go and check. A confident wrong value is worse than an admitted blank, because a blank is honest and gets dealt with, while a wrong value slides silently into everything downstream.
And the part that took me embarrassingly long to accept: "it isn't there" is a complete answer. It does not need to be improved into something that looks more like information.
The version you can paste in right now
All of it collapses into five lines. Put them in your assistant's instructions file, your saved prompt, your project settings, wherever your tool keeps standing rules:
- Before running anything that produces a result: say what you expect
and why, in one or two sentences. Then run it.
- Also say how I'd know if this were broken, and check that first.
- "Done" is not a result. Show me the output: the file, the rows, the value.
- Anything meant to warn me gets tested once by making it warn me.
- Look facts up instead of recalling them. "Not there" is a valid answer.
If five lines is four too many, keep the second one. It's the one that has paid for itself the most times.
Can you actually prove this helps?
Not the way I'd like to. I can't show you that it produces fewer invented facts, because I haven't run that experiment and I'm not going to claim a result I don't have.
But that isn't really the thing that hurts, is it. What hurts is the week you spend building on top of an answer that was wrong from the start. And that one is measurable, by you, this month, for free.
Keep a tally. Every time you have to walk something back, throw work away, or redo a decision because the result underneath it turned out to be wrong, put a mark down. One line, one date, one sentence on what the wrong thing was.
Do that for two weeks as you work now. Then add the two questions and do it for two more.
You're not counting how often the assistant sounds right, which is close to always and tells you nothing. You're counting how often it stayed right long enough to build on. That's the number that maps onto your actual time, and unlike a benchmark score it's yours, it's specific to your work, and nobody can argue with it.
My honest position: I have a mechanism I find convincing and an incident log where the disasters cluster hard on the occasions when nobody committed to anything in advance. That's evidence. It isn't a measurement, and I'd be doing the exact thing this post argues against if I dressed it up as one.
Where it doesn't help
Three limits, because I'd rather say them than have them said back to me.
It needs something checkable. If there's no observable outcome, "predict the result" turns into a confident-sounding preamble that flatters whatever comes next. When there's nothing to be wrong about, skip it.
A confident prediction can pull you toward it. Say a number out loud beforehand and there's pressure, on a model as much as on a person, to read the result as agreeing with it. The fix is to predict the observable, not the conclusion. "There should be roughly 400 rows, all dated after June" pulls much less than "this should confirm my theory."
It slows you down slightly. A few seconds per task. You get them back the first time it catches something, and you'll know exactly when that happens, because the prediction will be sitting right there next to the result, disagreeing with it.
If you try the five lines, I'd like to know which ones earn their place and which are cargo cult. I'm fairly sure at least one of them is.
Top comments (2)
The prediction step is useful because it forces the assistant to name the measurement before it can rationalize the result. I have had better luck when the predicted output includes a small falsifier too, something like the exact file, row count, or command result that would prove the run looked at the wrong thing.
Agreed, and I'd push it further: the falsifier is the load-bearing half. A prediction on its own cannot tell you the run looked at the wrong thing, because the prediction and the result can both be wrong in the same direction and agree with each other perfectly. That's what the second question in the post is for, and your version of it (name the exact file, the row count) is sharper than mine.
The thing I'd add is that what you pre-register depends on the shape of what you expect. When the output is a number, I write down the number and the derivation, not just the number. The derivation is the part worth falsifying: a value that lands close for the wrong reason costs more than one that misses, because nothing ever prompts you to go back and look at it.
I should be clear that I have no A/B on any of this, just a mechanism and an incident log. Have you hit a kind of task where the falsifier is genuinely hard to state?