The task looked routine. Update a comparison table on a live page — add a new property, adjust a few rows, push to production. The AI completed it, reported it done, and I moved on.
Two months later, a third-party analysis flagged an entity conflict: a property was listed in the wrong city. Not the wrong building or the wrong street — the wrong city entirely. One was 100+ miles from where the page said it was. The page had been live, indexed, and receiving organic traffic the whole time.
The page loaded fine. No build errors. No type errors. No 404s. Nothing indicated anything was wrong.
That's the confidence gap.
What happened
The AI was tasked with building a property comparison table from scratch. It needed names, locations, prices, and policies for each entry. Some values existed in the database. Others the AI inferred from surrounding page content and previously generated text.
It didn't label those inferences as guesses. It just wrote them. And then reported the task complete.
Here's what the error looked like — anonymised, but structurally identical to the real failure:
| Property | Correct city | What the page said |
|---|---|---|
| Property A | Morrison | Pueblo |
| Property B | Manitou Springs | Colorado Springs |
| Property C | Denver | Fort Collins |
Three properties. Three wrong cities. All live, all indexed, all quietly sending wrong signals to Google's entity model.
The specific mechanism: one property's city field was inferred from surrounding prose on the page — prose that had itself been written by a previous AI task. The error was already baked in before the table was built. When the AI read that text as context, it accepted it as fact, and propagated it forward into the new table.
AI wrote it wrong. AI read it back. AI wrote it wrong again.
That's not a hallucination — it's a feedback loop. And unlike a hallucination, it's internally consistent. Nothing on the page contradicts itself. The error is coherent. That's what makes it invisible.
Why the feedback loop matters
Google's entity model doesn't just read your content — it cross-references it. When a page describes a property as being near a specific landmark, but the location field places it 100 miles away, that's a geographic contradiction. The entity resolver flags it. That's a real SEO signal, and it works against you silently, the same way the error was introduced.
This is the part that makes the feedback loop worse than a one-time mistake. A single wrong fact might degrade one page. A wrong fact that gets read back and propagated spreads across the site — each new page adding another source of the same contradiction, compounding the entity signal problem.
Why confidence is the actual problem
Most AI failures are obvious. The code doesn't compile. The page crashes. The API returns an error. You see it immediately.
This failure class is different. The output is syntactically and visually correct. A human skimming the table would not notice unless they already knew the right answer. The AI's tone when reporting completion — "done, pushed" — is identical whether it verified the fact or invented it.
That's the confidence gap: the signal you'd use to detect a problem (confident completion) is the same signal the AI emits when everything is fine.
The gap is especially dangerous on commercial sites because:
- Content is published and indexed fast
- Errors compound across pages — wrong fact in one place gets read back and written into the next
- The business stakes are real — wrong location data affects SEO entity signals, user trust, and affiliate conversions
A developer making this mistake would usually notice when they went to verify the property existed. The AI skips that verification step entirely unless it's explicitly required.
The audit
After finding the first error, the right move was a full audit — not just fixing the one instance and moving on.
The process:
- Query the database directly for source-of-truth values on every property: name, city, address
- Compare each hardcoded reference on each page against the DB record — not against other page content
- Flag discrepancies before touching anything
- Fix only confirmed errors, one at a time, with a pre-push summary table every time
Across 40 pages audited, the full damage report:
| Error type | Count |
|---|---|
| Wrong city | 3 properties |
| Wrong name | 2 properties |
| Inconsistent rating | 6 instances |
| Total corrections | 9+ |
All of these had been live. None had triggered any automated alert.
The key discipline: the database is the source of truth, not the page. Auditing a page against itself finds nothing — it just confirms the error is internally consistent. You have to go upstream.
The process change
Going forward, the rule is simple: no property detail gets written without a verified source. If the city isn't in the DB record, you ask for it. You don't fill the gap.
Before every push to a commercial site, a pre-push summary table is required:
| File | Field | Before | After |
|---|---|---|---|
| ExamplePage.tsx | location | Old value | Verified value |
The human reviews that table before the commit happens. Not after.
This doesn't eliminate AI mistakes — it makes them visible before they go live. That's the only realistic bar. AI tools will continue to infer values from context when they don't have a verified source. The operator's job is to build a process that catches that gap before it ships.
What this means for operators
If you're using AI to build or maintain a commercial site, assume there are errors you haven't found yet. The absence of visible problems is not evidence of accuracy.
The most dangerous AI outputs are the ones that look right. Code that doesn't compile gets fixed immediately. Content that's factually wrong but grammatically perfect sits in production for months — getting indexed, getting read back, getting propagated.
Run your audit from the source of truth, not from the page. Build the pre-push review into the workflow before something gets indexed. And treat AI confidence as a neutral signal — it tells you the task completed, not that the output is correct.
The gap between those two things is where production errors live.
Top comments (0)