The property was called something plausible. It had a star rating, a price, a tidy paragraph of description, valid LodgingBusiness structured data, and a green "Verified" badge. It also had a Book Now button wired to a working affiliate link — the kind that pays a commission if someone clicks through and reserves a room.
The property did not exist. Not "closed," not "renamed." It was never real. Some fast, AI-assisted build had generated it whole — name, blurb, rating, amenities — and pointed it at a live payment rail.
It wasn't alone. There was a whole cohort of them, and the giveaway was almost funny: a dozen distinct "hotels" in different cities all sharing the same affiliate link. Different names, different descriptions, one destination URL. A forger who signed every painting with the same signature.
None of them were broken. That's the part worth sitting with. They passed every automated check I could throw at them. Valid types. Valid schema. No 404s, no build errors, no lint warnings. A validator confirmed they were well-formed, and being well-formed was the entire disguise.
Wrong implies there's a real thing being described
I'd spent earlier weeks fixing data that was wrong — a fact gone stale, an entity filed in the wrong city, a corrected value that never reached the page. Wrong is a solved kind of problem. Wrong means there's a true answer and the stored one disagrees. You audit, you compare against a source, you reconcile.
This was different, and it took me a beat to see why. You cannot reconcile a fabrication against a source, because there is no referent. The row doesn't describe a real thing incorrectly. It describes nothing, confidently. There's no fact to check because there's no subject.
So the first reframe: AI-generated seed data isn't a rough draft you refine. It's a forgery you have to detect — and its defining property is that it validates. Every instinct we've built says malformed data is the dangerous data, because malformed data is what breaks things. But malformed data announces itself. The genuinely dangerous record is the one that is perfectly formed and completely false, because nothing in your pipeline is designed to doubt a row that parses.
I deleted them. You don't refactor a forgery.
The quieter version of the same problem
The fake listings were the loud case. Underneath them was a quieter one that had spread much further, and it's the part that actually changed how I think.
Real listings had a policy field — a small piece of data recording whether a given property had actually been confirmed to permit a certain thing on-site. Most of them had never been confirmed. The field was empty.
Here is what the page rendered for an empty field: a green Verified badge. A "Yes" in the amenities grid. A line of structured data asserting the thing was permitted. A confident sentence in the server-rendered copy that a crawler reads before any of the interactive parts load.
Nobody decided to claim these things. There was no bad actor and no bad data. There was one line of code, repeated in spirit across the codebase, that read roughly: if this value is missing, treat it as true. A default. ?? true, or its cousin in four other files.
That is the second reframe, and it's the one I'd tattoo on a new engineer: a default is a claim. When you fill an unknown with the favorable value, you are not being helpful and you are not leaving it blank. You are asserting a fact you do not have. The empty field didn't stay empty on the way to the user. It was quietly promoted to a promise.
Absence rendered as a yes.
The missing value was "I don't know"
Once I saw it in one place I saw it everywhere, and they all traced to the same root cause. The data model had two states where it needed three.
There was true, there was false, and there was null — but null was never a state of its own. It was an input to be resolved, and every layer resolved it the same optimistic way. The card coerced it to yes. The badge coerced it to yes. The structured data coerced it to yes. The prerendered copy a crawler reads coerced it to yes. One missing fact became four confident assertions, none of which had any way to represent the honest answer.
Because there was no honest answer available. "Unverified" wasn't a value the system could hold. And a system that cannot represent "I don't know" does not fall silent about what it doesn't know. It reaches for the most flattering value in range and commits.
The fix was not clever. It was to add the value that was missing.
- A real, explicit state for unconfirmed — stored, not inferred.
- A three-way rendering instead of two: Yes, No, and Unconfirmed. The unconfirmed state gets its own muted styling — never the green of a confirmed yes, and just as importantly never the red of a confirmed no. "We haven't checked" is a different claim from "this is not allowed," and collapsing them is its own kind of lie, pointed the other direction.
- Every layer taught to defer to that state instead of defaulting past it. The card, the badge, the structured data, the static copy the crawler reads — all four now say "unconfirmed" together, or say nothing, rather than four independent optimistic guesses.
- Positive claims gated behind an actual confirmed value, so the system asserts a thing is true only when something actually backs it.
Give the model a way to shrug
The lesson generalizes past any one field or any one site. It applies anywhere a system holds facts it can't fully back — which is everywhere, and increasingly so as more of the initial data gets generated rather than entered by someone who checked.
We tend to think of honesty in a system as the absence of false statements — no bugs, no stale rows, a clean audit. But you can pass every audit and still have a system that lies, because the lies aren't in the data. They're in the defaults. They're in what happens to a value you never had.
Real honesty is the presence of "unknown" as a first-class thing: a value you can store, a state you can render, a claim your structured data is allowed to decline to make. If the only options your model offers are yes and no, then every gap in your knowledge silently becomes whichever one flatters you — and the extreme end of that gradient is a hotel that doesn't exist with a working button to book it.
Give the model an explicit way to say "I don't know." Otherwise it will nod at everything, and you won't find out where until someone audits the room and discovers half of it was painted by the same forger.
Further reading — the adjacent failure modes: why AI tools produce confident silent errors on production sites, and why a fallback chain hides missing data by design. This post is the layer under both: why the gap gets filled with a lie in the first place.
Originally published on tedagentic.com.
Top comments (0)