I had a model refactor part of a backtesting engine last month. Gave it function signatures, call chain, test suite, the works. Output looked solid, tests passed. Then I renamed a variable and watched one assertion go sideways. Turned out the model had wired that variable to the wrong thing because I never explained what it actually meant. I added one sentence of context. Regenerated. Fixed.
Same week, different mood, I asked a model whether it understands sadness. Got back something careful and well-formed. It can recognize patterns of sadness in text, it said, and it can offer supportive responses, but it is not sure whether that constitutes understanding. Reads well. I genuinely could not tell whether it understood sadness or whether it had reproduced, very fluently, the statistical shape of how humans write about sadness. Its training set has seen more breakup posts than any therapist alive.
Then I looked at a multi-agent trading framework on GitHub. Tens of thousands of stars. Five LLM agents debating each other into a trade signal. The backtests looked spectacular.
Three interactions, three completely different feelings. The first one, the model genuinely helped. The second one, I am not sure what happened. The third one made me nervous.
So what is "hallucination"? All three of those? We use the same word for a model inventing an API method that does not exist and for a model confidently dating a military strike, and then we act surprised when "fix hallucination" does not converge on a solution. Calling a head cold and a tumor "feeling unwell" does not help anyone pick a treatment.
One question sorts them out
Here is the test I started applying. Can the missing context be obtained?
The backtesting bug was a yes. The context was finite, sitting right there in the codebase, verifiable against a test suite. I supplied the missing piece, the model corrected itself. This is the hallucination that chain-of-thought, bigger context windows, and multi-turn dialogue actually help with. The information exists. Only question is whether it reached the model.
The sadness question is different. The model's training data has mountains of emotional writing. It can output something that looks deeply informed. But did it understand, or did it reproduce the shape of understanding? I cannot verify which thing happened, and neither can it. The missing context here is not a file I forgot to attach. It is a fact about the model's interior that no one has an instrument for.
Then the predictions. Tomorrow's strike date, tomorrow's closing price, tomorrow's anything. The missing context is not "withheld from the model." It is not accessible to anyone at the moment you ask. Not to the model, not to you, not to the analyst you would pay good money to hear from. This is not a context gap you can close by writing a better prompt. The information is not reachable from where any of us are standing.
Three kinds of missing context:
- First kind: context exists, can be supplied, output is verifiable. Fixable.
- Second kind: output is often useful, but the underlying question (did it "understand"?) is not verifiable. Usable with eyes open.
- Third kind: critical context does not yet exist. Model can only pattern-match from history. Not fixable by any prompt.
The benchmarks are teaching models to bluff
A paper from OpenAI and Georgia Tech last year (Kalai et al., "Why Language Models Hallucinate," arXiv: 2509.04664) landed on something I had not thought about clearly. They audited the leaderboards everyone optimizes against. GPQA, MMLU-Pro, SWE-bench. Nine out of ten penalize saying "I don't know." You score zero for abstaining. You score something for a confident guess. Train a model under that regime and it learns the only lesson the scoreboard teaches: bluffing beats honesty.
(We built a test that rewards bluffing and then act surprised that the models bluff. I don't know what we expected.)
Their fix makes sense for the first kind: bolt confidence thresholds onto scoring, penalize wrong answers harder than abstention, let the model learn that shutting up is sometimes the right move. A model that says "I'm not certain what this code path does" instead of confidently inventing an explanation? I would ship that today.
But notice where the fix stops. It assumes there is a right answer the model could have given if only it had been braver about saying "not sure." For predictions, there is no right answer in the model to be brave about. Abstention is the only correct response to every prediction query, and a model that abstains on every prediction is not a product anyone is going to pay for. So the market pressure runs the other way.
A separate paper (Xu et al., "Hallucination is Inevitable," arXiv: 2401.11817) puts a floor under this. Using learning theory, they prove an LLM cannot learn every computable function. Deployed as a general-purpose answer machine, it must hallucinate somewhere. Sounds dire. Actually just structural. The base next-token generator has no built-in "do I understand this well enough to answer?" module. The probability distribution over next tokens is always non-empty, so there is always a next token, so the generator always speaks. You can bolt a confidence gate on top, a verifier, a reasoning loop, whatever outer scaffolding you like. That is where the incentive fixes live. The thing being scaffolded has no internal stop.
A human has a real internal state called "I don't understand." That state blocks output. You stop talking because you genuinely have nothing. The model has a probability distribution conditioned on the input, and it samples. That is the entire transaction.
Chain-of-thought helps on the first kind, where rephrasing a few times means at least one reasoning path avoids the gap. On predictions it produces a hundred well-reasoned, mutually contradictory fabrications, because the answer was never in there. CoT is a person with no plan, told to draw up a plan first, drawing the plan with no plan.
Knowing the line is there is not the same as being stopped by it
On the emotional questions, the training has actually done some work. Ask a current model whether it feels things and it will hedge. It has been taught that a line exists here.
Watch what happens next though. It does not stop talking about emotions. It answers at length, fluently, in a way that reads as deeply informed, and tacks a caveat onto the end. Recognizing the line and being constrained by the line are two different things. The model knows the cliff is there. It walks right up and keeps narrating.
That is the second kind, where at least step one happened. On predictions, the model has not even taken step one. It does not flinch before dating a war. Push harder and it gets more specific.
A newspaper tested this
In early 2026 the Jerusalem Post ran an experiment. They asked four language models when the US would attack Iran and then pressed each one for a specific date.
Four models, same public information, four incompatible answers. Grok said February 28th. ChatGPT said March 1st, then moved itself to the 3rd. Gemini drew a window of March 4th to 6th. Claude held out longest and settled on the weekend of the 7th. Four confident answers, mutually exclusive, each with a reasoning chain that read as careful and precise. At most one could be right, and nothing in any chain told you which.
The thing the Post itself flagged: the harder the models were pushed, the more specific the answers got, even though nothing in the real world had become any clearer.
Pressure raised the resolution of the output and left the accuracy exactly where it was. The model did not get closer to the truth. It got more detailed about something it could not know.
Look at the four dates through the lens of training data. Every answer was reasoned from the same templates the distribution is thickest on: rational game theory, relative military power, the rhythm of past confrontations, diplomatic deadlines. That is why they clustered in the same ten-day window and only disagreed on which day. They were all extending the same consensus pattern. But a strike date gets decided by the part of reality that departs from the template. An internal political call, an intelligence window, a single decision-maker's nerve on a Tuesday morning. None of that is in the distribution, so none of it is in the output.
The mode trap
So what is the model actually doing when it "predicts"? Extending the highest-frequency pattern in its training data. Not "what is most likely to happen next" but "what most commonly followed situations that looked like this." Those two phrasings sound identical. They are constantly, importantly different.
In any competitive domain, trading, geopolitics, anything adversarial, the pattern everybody can already see is by definition not an edge. The model hands you the consensus. The consensus is the mode. The mode is what the training distribution piles up on. So in exactly the domains where a prediction would be worth something, the model's prediction is the single least useful thing in the room. It is handing you what everyone already knows, formatted nicely.
Markets make this concrete. Consensus says gap up. So a large player who wants size at a better price engineers a gap down, and everyone who traded the consensus gets taken out at the bottom. Model stands with the crowd every time, because the crowd is the mode. If standing with the mode paid, there would be far more rich people walking around.
Hallucination to the power of n
This is the lens that finally named what bothered me about those multi-agent trading frameworks. Five or six LLM agents, fundamentals, technicals, sentiment, risk, a final arbiter, debating each other into a trade signal.
None of them has tomorrow's data. Five agents conferring does not manufacture predictive information. It manufactures five angles on yesterday.
Then it compounds. Agent A produces a confident read off stale data. Agent B is supposed to check A, but B has no live data either. All B can actually verify is whether A is internally consistent. And internal consistency is the signature property of confident hallucination. Wrong, but it hangs together. Reads great. B blesses A's coherence and mistakes it for accuracy. Every additional agent ratifies the fabrication beneath it. This is not error averaging out across a committee. This is error compounding up a stack. Peer review, except none of the peers have been outside.
Why do the backtests look great? Because backtesting runs on history. History has already happened, which makes it a first-kind problem with complete context, and the first kind is exactly where these models shine. The gorgeous equity curve is the model being brilliant at the one thing it can do, dressed up as evidence for the one thing it cannot.
It is not a usage problem
I was going to end with a tidy table. Three kinds, three recommendations, here is your checklist. Deleted it. Felt false.
The problem is not how to use the tool. It is what the thing on the other end actually is, and that changes depending on which side of the line you are standing.
First kind, it is a tool. Supply context, get output, verify against reality. Clean relationship.
Second kind, it is a mirror. What comes back is humanity's writing about understanding, feeling, consciousness, reflected at you. Often useful. But you are deciding what is substance and what is just your own field talking back, and the model cannot help you sort that.
Third kind, do not call it collaboration. It does not have the thing you need, and you cannot hand it over, because the thing has not happened yet. Two of you "analyzing" tomorrow's price is two students who never saw the exam comparing answers with total confidence.
The unsettling part: a single question can sit across the boundary. "How will this architecture perform once it's live" looks like pure engineering, but "once it's live" smuggles a prediction inside it. The model hands back a rigorously engineered analysis with fabricated forecasts laced through, and from the prose alone you cannot tell which clauses are grounded and which are pattern extrapolation. That is the most common situation in real work. Not a clean third-kind question you can refuse, but a first-kind question with a third-kind passenger.
Maybe I am cutting this too cleanly. The borders are not crisp. Where does writing a poem go? Context is not missing, nothing is being predicted, but whether it is any good is a judgment call. Where do moral questions land, where the model draws on ethical consensus that is itself drifting? I have not figured that out.
What gets me is the people, not the taxonomy. Senior scientists at frontier labs wiring LLMs into trading pipelines. A national newspaper using them to forecast a strike and running it as reporting. Funds writing "AI-assisted analysis" in the research note. These are not people who do not get the technology. Many of them know the architecture better than I do.
So either my line is in the wrong place, or something is going on that I am not seeing. I genuinely do not know which. Maybe capability shifts the line. Maybe some of them found a way to pull real signal out of the third kind that I have not learned.
But here is the question worth more than another round of "just prompt it better": when the people who understand these systems best are working on the far side of the line, how do you tell the ones who discovered the line is not really there from the ones who are pretending it is not?
I do not have that answer. I only know that until this field treats that as the actual question, hallucination will keep getting filed as a prompting problem with a prompting fix. It is not one.
References
- Adam Tauman Kalai, Ofir Nachum, Santosh S. Vempala, Edwin Zhang. "Why Language Models Hallucinate." arXiv:2509.04664, 2025. https://arxiv.org/abs/2509.04664
- Ziwei Xu, Sanjay Jain, Mohan Kankanhalli. "Hallucination is Inevitable: An Innate Limitation of Large Language Models." arXiv:2401.11817, 2024. https://arxiv.org/abs/2401.11817
- The Jerusalem Post. "Grok predicted Feb. 28, then Israel and the US struck Iran." 2026-02-28. https://www.jpost.com/middle-east/iran-news/article-888274
Find me on GitHub | Substack | StratCraft
Top comments (0)