I run a pipeline that generates explainer articles. LLM in the middle, structured output, published in several languages. It had been running for a while and the articles looked good: clean layout, a chart, and near the top of each one a confident little box with a statistic. Something in the shape of "68% of people never change the default." A number, a source, an authoritative ring to it.
Not one of those numbers had been researched. The pipeline had never looked up a single statistic in its life. It asked the model for a number and printed whatever came back.
I did not find this through a clever eval. I found it while cleaning up something unrelated and actually reading the prompt.
The field that forced a lie
The output schema had a required field. statistic.text and statistic.source, described in the prompt as an "eye-catching stat" for the top of the article. Required. Every article had to have one.
The prompt also, helpfully, told the model what to do when it did not have a real number. It said to round to a safe order of magnitude. And it said to strip the year off the source, so the article would look evergreen instead of dated.
Read that back slowly. The instructions were: always produce a statistic, make up a plausible magnitude if you have to, and remove the one piece of metadata that would let anyone check it. That is not a prompt that occasionally allows a hallucination. That is a prompt that requires one, every single time the model does not happen to know a real figure.
So it produced them, confidently, in every language, each wearing a real-sounding source: a named institute, an industry association, a government statistics office. None of it had been looked up when it was written. This was content people actually act on, which is exactly the category where being wrong is not a rounding error.
There was a second engine doing the same thing in the chart code. The block that generated the data visualization asked the model for "actual statistics from official sources" and, to be helpful, gave "Bureau of Labor Statistics" as an example. So the chart invented its own numbers too, separately from the stat box.
Why "just tell it to be honest" does not work
My first instinct was the wrong one. Add a line to the prompt: only use real statistics, cite a verifiable source, do not make anything up.
That does nothing, and it is worth being precise about why. A language model cannot tell a hallucinated number from a remembered one. From the inside they feel identical. When you ask it to "only cite real sources," it agrees, sincerely, and then hands you a real-sounding source with the same sincerity whether or not it exists. Honesty is not a knob you can turn from inside the thing that has no way to check itself.
The bug was never linguistic. It was structural. A required field forces fabrication whenever the model lacks a real value, and no amount of polite prompting changes that, because the field is still required and the model still has no number.
Once you see it that way the fix is obvious and slightly boring.
- The statistic field became optional. If there is no real number, the section is simply skipped when the page renders. Nothing to fill, nothing to invent.
- A number is now allowed only if it is backed by a primary source or an official table AND covered by a real source URL in the article's own sources list. Otherwise the model is told to say it qualitatively: "in most cases," "within a short window," without a fake decimal.
- The chart block lost its "use official statistics" instruction and its Bureau of Labor Statistics example. It now draws only values that already appear in the article text, and if there are none, it draws a qualitative chart of steps or stages instead.
- The quality check that scored each article used to deduct points for a missing statistic. That was rewarding the exact behavior I was trying to kill. Now it deducts nothing for an absent number and flags a source cited from memory as critical.
The difference showed up immediately. Where the model used to reach for an official-sounding report that did not exist, it now points to a primary source that does, or it says nothing at all. A citation it can stand behind, instead of a plausible one it made up.
Then I went through the 28 live articles by hand. Between them they carried 35 of these statistic boxes. I removed 27 that were fabricated or could not be traced to a source, and kept 8 that turned out to be genuinely backed by a primary source or an official table. Four of the 35 were close enough to plausible that I checked each one against a real search. Three were wrong and went: a claimed 16.8 million was really 6.2 million, another had no source I could find, a third was overstated. One was nearly right, so I corrected it to the exact value and kept it. That is the whole ledger, 27 out and 8 kept, and it is not flattering. An invented source is worse than no number at all, so the rule was simple: if it cannot be traced, it does not ship.
The same bug wearing a different hat
While I was in there I hit the identical mistake in a completely different place, and that is the part worth generalizing.
I had moved the cheap, mechanical calls in the pipeline over to a smaller model to cut cost. Not the main article text, which stays on the strong model, but the scaffolding: the layout code, the SVG, the chart data. That change cut the per-article cost roughly in half, and I was happy with it.
One of those mechanical calls rewrites the whole page component and needs a lot of output tokens. The small model caps out at 8192 output tokens. The call asked for more. The API did not error. It returned HTTP 200, a perfectly successful response, with a quiet finish_reason: length and the code cut off in the middle of a JSX tag. The build died several minutes later with "Unexpected token ArticleLayout," an error that points nowhere near the actual cause.
Same shape as the statistics bug. A property I could have checked mechanically, the output fitting inside the token limit, was instead assumed to hold because the prompt implied it should. So the failure was silent until it surfaced somewhere unrelated and expensive to debug.
The fix was the same kind of boring. Asking for more tokens than the model can produce now throws a config error before the call is even made. A truncated response now fails the run hard instead of passing it downstream. And I stopped routing that particular call to a model that cannot fit the job.
The actual lesson
Both bugs came from the same habit: asking the model, through the prompt, to guarantee something the harness could have enforced.
A prompt is a request. It is not a guarantee. Anything that can be checked in code, that the output must fit in the token limit, that a cited number must trace to a real source, that a required section must have real content or no section at all, has to be enforced in the code around the model, not requested inside it. The moment a mechanically verifiable property lives only in the prompt, you have not prevented the failure. You have just agreed not to look at it.
The uncomfortable version, for anyone shipping LLM output to real users: your model will fill any blank you make mandatory. If your schema requires a statistic, it will produce a statistic, real or not. If your prompt rewards a number, you will get numbers. The behavior you are seeing is not the model being dishonest. It is the model doing exactly what your structure told it to do. Fix the structure.
Top comments (0)