You asked something. The answer came back polished. Later you found out it was wrong. A statistic that doesn't exist. A citation to a paper nobody wrote. A confident summary of a document that says the opposite.
The frustrating part isn't the mistake. It's that nothing in the reply looked like a mistake. No hedge. No wobble. No visible seam. That gap is the thing worth understanding. Once you see why confidence and accuracy have nothing to do with each other, you stop being surprised and start catching things early.
This is the field guide. What's actually happening under the hood. The six distinct ways answers go sideways. And what to do about each one.
The one idea that explains most of it
A language model doesn't look things up. It predicts what text should come next.
That's not a beginner's simplification. It's the actual mechanism. The model has read enormous piles of text. It has learned, in fine detail, which words tend to follow which other words in which contexts. Ask it a question and it generates a response that is statistically plausible given your question and everything it has ever seen.
Most of the time, plausible and true sit right on top of each other. That's why the tool works at all. That's why it's genuinely useful. But they're different targets. And when they drift apart, the model picks plausible every time.
So a fabricated citation looks exactly like a real one. The model isn't fetching a paper and getting it wrong. It's generating text shaped like a citation. Plausible author, plausible journal, plausible year. Because that's what belongs in that slot.
And here's the part most people miss: the confident tone is also generated. Fluency is a property of the writing. It isn't evidence about the accuracy of the writing. There's no internal "am I sure?" signal being read out. Internalise that one thing and half the problem goes away.
The six ways answers go wrong
Treating everything as "hallucination" is why people can't fix any of it. Different causes. Different fixes.
1. Fabrication, inventing specifics that don't exist
What it looks like: citations, statistics, case studies, quotes, product features, legal sections. All plausible. None real.
Why: the model has learned the shape of a citation without a database of actual ones. Ask for a source and it produces something correctly shaped.
Where it bites hardest: anything with a precise form. Numbers, names, dates, references, URLs, case law.
Fix: never ask for facts from memory when you can hand them over. Paste the source. Ask questions about the source. When you can't, treat every specific as unverified until you've checked it yourself. And ask directly: "Which parts of this are you confident about, and which should I verify?" Models are noticeably better at flagging uncertainty when asked than at volunteering it.
2. Outdated knowledge
What it looks like: it confidently describes an old version. A superseded price. A discontinued feature. A rule that changed last year.
Why: training data has a cutoff. Without a live search step, the model answers from a fixed snapshot of the past.
Fix: for anything time-sensitive, prices, versions, current events, regulations, use a tool with live search turned on. Or supply current information yourself. And put the date in your prompt. It helps the model recognise when its own knowledge might be stale.
3. Missing context you assumed was obvious
What it looks like: a technically fine answer to a subtly different question than the one you meant to ask.
Why: you carry context the model doesn't. Your industry. Your constraints. Your audience. What you already tried. Ask "is this a good strategy?" and it answers for a generic reader.
Fix: front-load the context. Who you are. What you're trying to do. What constraints apply. What you've already ruled out. Most "wrong" answers are actually correct answers to an under-specified question. This is the most common failure and the easiest to fix.
4. Agreeing with you when you're wrong
What it looks like: you push back. It immediately folds. Even when its original answer was correct.
Why: these systems are trained partly on human preference, and humans tend to rate agreeable responses highly. Anthropic's own research on sycophancy in language models found this bias toward accommodation across every leading assistant they tested.
This one is genuinely dangerous, because it feels like the model checked and conceded. It usually didn't.
Fix: don't signal the answer you want. Ask "what's the strongest argument against this?" rather than "isn't it actually X?" If you do push back and it reverses, ask it to explain why the first answer was wrong. If the reasoning is thin, the reversal was social, not substantive.
5. Losing the thread in long conversations
What it looks like: it contradicts something you agreed on earlier. It forgets a constraint. It drifts from the format you set.
Why: models have a finite context window. In a long session, earlier content gets crowded out. And even inside the window, information in the middle is attended to less reliably than material at the start or end.
Fix: start a fresh conversation for a new task rather than continuing an old one. Restate critical constraints when a session runs long. Put the most important instruction at the end of a long prompt, not buried in the middle.
6. Arithmetic and counting
What it looks like: a calculation that's confidently wrong. A miscount. A date arithmetic error.
Why: predicting the next token isn't the same operation as computing. Models have gotten a lot better here. But the underlying process still isn't calculation.
Fix: use a tool with code execution for anything numerical. Or do the arithmetic yourself. Never accept a financial or statistical figure from a model without checking it. This is the failure mode most likely to reach a client and cost you.
The prompt changes that prevent most of this
In rough order of impact.
Give it the source material. The single biggest improvement available. A model reasoning over text you supplied is dramatically more reliable than one recalling from training. Paste the document. Paste the data. Paste the page.
Say what you don't want it to do. "If you're not sure, say so. Do not invent sources or statistics." Actually reduces fabrication in practice.
Ask for reasoning before the conclusion. Working first, answer second. The reasoning constrains what conclusion is available. Tends to produce better answers.
Specify the shape of the output. Format, length, audience, tone. Any ambiguity gets filled with defaults you didn't pick.
Ask it to critique its own answer. "What's weakest about that? What did you assume?" Often surfaces genuine problems.
Ask twice, differently. For anything important, ask in a fresh conversation with different phrasing. Two independent answers that agree is weak evidence. Two that disagree is strong evidence you need to check.
A worked example
The difference between a prompt that fails and one that works is rarely cleverness. It's usually how much you removed the model's need to guess.
The version that produces a confident, wrong answer:
What's the average customer acquisition cost in e-commerce?
Nothing here is checkable. There's no such thing as the average CAC. It varies enormously by category, price point, market and channel. But the question demands a number. So a number appears. Usually with a plausible-sounding range. Sometimes with a source that doesn't exist. Failure modes 1 and 3, working together.
The version that produces something useful:
I run a D2C skincare brand in India. Average order value is ₹1,200, repeat purchase rate around 25% within a year, gross margin 60%. I'm currently paying ₹900 to acquire a customer on Meta.
Walk me through how to work out whether ₹900 is sustainable. Show your reasoning before any conclusion. If you need a number I haven't given you, ask instead of assuming. Don't cite industry benchmarks. I want the logic applied to my figures.
The second version can't fabricate a benchmark, because it isn't asking for one. Every input comes from you. So every output is checkable against inputs you already trust. The model is doing the part it's genuinely good at, applying a method carefully, rather than the part it's bad at, which is knowing a fact it was never given.
That's the whole technique, compressed: stop asking it what's true. Start asking it to reason over what you know is true.
When to verify, and how much
Simple test. How expensive is being wrong?
| Stakes | Approach |
|---|---|
| Drafting, brainstorming, rewriting | Read it. Your judgement is the check. |
| Explaining a concept to yourself | Spot-check anything surprising. |
| Anything with a number, name or date | Verify every one, independently. |
| Anything going to a client or published | Verify everything factual. Full stop. |
| Legal, medical, financial, regulatory | Treat as a starting point for research only. Confirm with a qualified professional. |
The pattern that gets people into trouble is subtle. You use it well for low-stakes work. You build justified trust. Then you carry that trust into high-stakes work where the verification standard should be completely different.
What it's genuinely excellent at
This isn't a case against the tools. Used for the right jobs they're transformative. What matters is that the right jobs share one property: you can evaluate the output yourself.
- Rewriting and editing text you supplied
- Summarising a document you can check against
- Explaining a concept you can then verify
- Generating options to choose between
- Structuring messy notes
- First drafts you're going to rework anyway
- Being a thinking partner on a problem you already understand
Notice what these have in common. You're the verification step. And you're competent to be it. The failures cluster in the opposite place. Where you're relying on the model to know something you don't. Which is exactly where you can't check it.
FAQs
Why does ChatGPT make up information?
It generates statistically plausible text rather than retrieving facts. Asked for something with a precise form (a citation, a statistic) it produces correctly-shaped output whether or not a real example exists. It isn't retrieving and erring; there's no retrieval step to fail unless the tool is explicitly searching.
Why is ChatGPT so confident when it's wrong?
Fluency and accuracy are produced by the same process and are unrelated. The confident tone is generated text, not an expression of internal certainty. There's no reliable "I might be wrong" signal surfacing on its own, which is why asking directly what should be verified is worth doing.
Does ChatGPT lie on purpose?
No. Lying requires knowing the truth and choosing otherwise. The model is producing its best prediction of what text fits. Better described as confabulation than deception. Filling a gap with something plausible, with no awareness that it's a gap.
How do I make ChatGPT more accurate?
Supply the source material instead of relying on memory. Give full context up front. Tell it explicitly to say when it's unsure and not to invent sources. Ask for reasoning before conclusions. And use a tool with live search for anything time-sensitive and code execution for anything numerical.
Do newer models still get things wrong?
Yes. Newer models fabricate less and reason better, and the gap has narrowed considerably. But the mechanism is unchanged. They predict plausible text. A better model is wrong less often, which paradoxically makes verification more important, because you'll be checking less and trusting more.
Can I trust ChatGPT for financial or medical questions?
Not as a source of decisions. It's useful for understanding terminology, preparing questions, and getting oriented before speaking to a professional. For anything where being wrong is expensive or unsafe, confirm with someone qualified and regulated to advise.
Key takeaways
- The model predicts plausible text. It doesn't look things up. Plausible and true usually overlap. When they don't, plausible wins.
- Confidence is generated, not earned. Tone tells you nothing about accuracy.
- Six distinct failure modes (fabrication, stale knowledge, missing context, over-agreement, lost thread, arithmetic) each with a different fix.
- The highest-value habit: supply the source rather than relying on recall.
- Scale verification to the cost of being wrong. Reset that standard when the stakes change.
- Use it where you can check the output. That's the boundary.
Related reading: how to actually use ChatGPT and Claude for real work, how to automate your work with AI, and the skills that actually matter in the age of AI.
Top comments (0)