Info Inlet

Posted on Aug 19

Your Website Is Telling ChatGPT You Don't Exist

#webdev #seo #programming #ai

Our iOS app went live on the App Store on a Tuesday. We build Xenition, an AI workspace — web, desktop, and now, finally, phones.

On Wednesday someone asked ChatGPT whether we had an iOS app. It said no. Not "I'm not sure" — a confident no, with a tidy little breakdown:

Android: live on Google Play
iOS: coming soon for iPhone and iPad
Desktop: available for macOS, Windows, and Linux

Every line was accurate as of six months ago. And it wasn't hallucinating. It was quoting our own website, correctly, from a page we had forgotten was making a promise.

I'm going to use our own files for every example below, because those are the ones I can quote exactly rather than paraphrase. None of it is product-specific — if you ship anything, you have this bug too.

That is the bug I want to talk about, because I don't think it's a marketing bug. It's a state management bug, and it has all the properties of the worst kind: it's invisible, nothing in CI catches it, and the blast radius grows every time you ship something.

Marketing copy is state, and nobody treats it that way

Here's the thing that took me embarrassingly long to see.

We are careful with state. Feature flags get cleaned up. Migrations are ordered. Config gets validated at boot. If a constant in the code goes stale, something eventually throws.

But a sentence on a landing page that says "coming soon" is also state. It's a claim about the world that was true when it was written and becomes false at some specific moment — a moment that happens somewhere else entirely, like an App Store review queue. Nothing links the two. No type system knows the sentence exists. The test suite is completely silent, because the page renders perfectly. It's rendering a lie, beautifully, with correct spacing.

The failure mode is specific: the more successful you are, the more of these you have. Every launch turns some "coming soon" into a falsehood. Every deprecation turns a feature list into a fib. And each one sits there being confidently wrong to every visitor and every crawler until a human happens to reread that paragraph.

So when I went looking, I didn't look for "the download page." I grepped for claims.

The audit: grep your repo for every place that states a fact

I asked one question: where does this codebase assert something about what our product currently is?

For a launch, the search terms are boring and effective:

# the promises
grep -rni "coming soon\|not yet\|in beta\|waitlist\|early access" src/ public/

# the platform claims
grep -rni "ios\|android\|app store\|google play\|desktop app" src/ public/

# the counts that drift
grep -rnE "[0-9]+\+? (users|customers|templates|integrations|languages)" src/ public/

That last one is worth its own paragraph. Numbers in marketing copy are the most reliably stale things in any repo. "200+ integrations" was true once. Nobody decremented it when three got removed, and nobody incremented it when twelve got added, so it's now wrong in both directions simultaneously.

For one launched feature, the grep found four places. I expected one.

#	Where	Who reads it
1	Download page, iOS card	Humans
2	Homepage, "All devices" feature card	Humans
3	Homepage FAQ, "Is there a mobile app?"	Humans
4	`llms-full.txt`	Machines

Three of those are ordinary UI. The fourth is the one that had actually been talking to ChatGPT behind my back.

The file you probably haven't written

If you haven't run into it yet: llms.txt is a convention — a markdown file at the root of your domain describing what your product is, in prose, for language models. Think robots.txt, but instead of "here's what you may crawl" it's "here's what we actually are." Many sites ship a short llms.txt and a longer llms-full.txt.

The pitch is straightforward. An LLM answering a question about your product has three options: reconstruct it from training data that's months old, scrape your marketing site and try to separate substance from hero-copy, or read a document you wrote specifically to be read this way. The third one is the only one where you have any influence at all.

Here's the part that bit us. Because this file is prose, and because it's genuinely useful, it's the densest concentration of factual claims in the entire repo. Ours had two lines about mobile. Both were pre-launch. So the single most authoritative machine-readable description of our product was also the most confidently wrong.

Nobody reviews that file at launch. It's not in the design mock. It doesn't show up in QA. It renders on no page. It is pure, unwatched state — and it's the file specifically written for the systems that answer questions about you at scale.

Fix the marketing pages and forget this one, and ChatGPT keeps saying no.

Machine-readable pages need real URLs, not identifiers

A small, concrete thing worth stealing.

Our old line said the Android app existed and gave the package name, com.infoinlet.xenition. Technically that's the correct identifier. Practically, it's useless to a language model trying to help someone install your app — it can't hand a user a package name.

The new version gives full URLs for both stores:

- Mobile: native apps on both stores — iOS at https://apps.apple.com/app/id6790250577
  and Android at https://play.google.com/store/apps/details?id=com.infoinlet.xenition

Now the answer comes back with something clickable in it. Same fact, dramatically different usefulness.

While we're here — for the App Store, use the bare form:

https://apps.apple.com/app/id6790250577    ✅
https://apps.apple.com/us/app/xenition/id6790250577    ❌

Apple redirects the bare form to the visitor's own storefront. The /us/ version, copy-pasted straight out of App Store Connect, sends everyone outside the US to a "not available in your country" page — for an app that is perfectly available in their country. If an LLM quotes that link, it quotes it to the whole world.

Your translations are lying too, and your tooling says everything is fine

Now the part I don't have a clean answer for.

Two of those four claims live in i18n strings. We ship 35 locales. Every one of them had that string translated — properly, by a human or a good pipeline — back when the claim was true.

So the moment I changed the English, all 35 translations became false. And here's the vicious part:

Coverage tooling reports 100%. Our i18n pipeline — like every i18n pipeline I've used — compares locales against the source and reports what's missing. A key with a translated value is a key that's done. It has no idea the English underneath it changed meaning. From the tooling's point of view, nothing happened.

Missing translations are visible: you see English text in a Japanese UI and you file a bug. Stale translations are invisible: you see fluent Japanese making a claim that stopped being true in August.

The fix is a content hash. Store the hash of the English string that each translation was made from, and a translation whose source hash no longer matches is stale, not done:

{
  "faq.mobileApp": {
    "value": "モバイルアプリはありますか？...",
    "srcHash": "a3f9c1"
  }
}

One extra field, and drift becomes detectable instead of theoretical. We hadn't done this. We're doing it now.

The immediate mitigation, if you're mid-launch and can't retranslate 35 files today: make sure a stale translation degrades rather than breaks. When I needed those FAQ answers to carry a real link, I didn't change the shape of the i18n object — a new field would have been missing in all 35 locales. Instead the plain-string answers now accept inline [label](url) markdown, and a small component renders it as an anchor. An answer without that markup renders exactly as it always did. So the 35 locales show old copy — bad — instead of showing raw markup or losing the paragraph — worse.

Deploying the fix does not fix the answer

This is the expectation I had to reset, and it's the one most people get wrong.

We merged, CI deployed in three minutes, and I verified the live file with curl. Correct. Then I asked ChatGPT again.

Same wrong answer.

There are three completely separate systems here and only one of them you control:

1. Training data. If the model is answering from memory, your website is irrelevant. That corpus was frozen months ago and nothing you deploy this week touches it. This resolves on the next training run, which is not a timeline you can plan around.

2. The search index. If the assistant browses, it usually goes through a search index rather than fetching your origin. The index has a cached copy from whenever it last crawled. Your deploy is live and the index is stale, and the gap is days to weeks.

3. Your origin. Correct within three minutes of merge. Congratulations, you fixed the one link in the chain that was already fastest.

Which means the useful move after shipping a fact-fix is not to keep refreshing your own site. It's to get the crawlers back:

IndexNow / Bing Webmaster Tools — submit the changed URLs. Several assistants browse through Bing, so this is usually the highest-leverage single action.
Google Search Console → URL Inspection → Request indexing, for the Google-backed side.
Check robots.txt isn't the actual problem. Name the AI crawlers explicitly rather than relying on the wildcard — several check for their own user-agent before deciding — and if you use content signals, make sure ai-input (use in AI answers) is set the way you think it is. It's an unforced error to fix your copy and stay blocked.

And a diagnostic that saves a lot of confusion: ask the assistant to browse the URL explicitly — "fetch xenition.com/download and tell me what it says about iOS." If the forced fetch gives the right answer, your site is fine and you're waiting on an index. If it's still wrong, you have a real problem — a caching layer, a robots rule, or a page that needs JavaScript to say anything at all.

That last one deserves flagging: if your factual claims only exist after hydration, a crawler that doesn't run your JS sees an empty div. A static llms-full.txt sidesteps that entirely, which is a large part of why it's worth having.

What still doesn't work

Being honest about the edges:

llms.txt is a convention, not a standard. Adoption is uneven and nobody publishes which crawlers actually consume it. It costs you one file to be readable by the ones that do, but I can't show you a number that proves it worked.
Nothing tests a claim. I have no good CI check for "this sentence is still true." A grep for coming soon in a PR diff is a start, but it can't catch "the fastest way to X" going false. The real fix is procedural: a launch checklist that includes "grep for claims," and someone whose job it is to run it.
Content hashing catches drift, it doesn't translate anything. You still need 35 translations. It just means you know you need them.
Training data is a closed door. For the specific question "does this product have an iOS app," a model answering offline will be wrong for months and there is nothing on your side to deploy about it.

The checklist

If you're launching something this quarter:

Grep for claims, not for pages. coming soon, not yet, beta, waitlist, platform names, and every hardcoded count.
Check llms.txt / llms-full.txt in the same pass. It's the densest set of claims you own and the least likely to be reviewed.
Put real URLs in machine-readable files, not identifiers. Bare App Store links, not the /us/ ones.
Assume your translations are now stale. Add a source-hash field so next time the tooling tells you instead of a user.
Make missing i18n degrade to old copy, never to broken markup.
After deploying, submit the URLs for recrawl. Your origin was never the slow part.
Verify by forcing a fetch, not by asking the assistant what it remembers.

None of this is difficult. It's just work nobody has assigned to anyone, which is exactly why your site is currently telling a language model something about your product that stopped being true a while ago.

Go grep for coming soon. I'd bet a coffee you find something.

If you've built an actual CI check for stale factual claims — not just a keyword grep — I'd really like to hear about it. That's the piece I don't have.

I work on Xenition — one AI workspace for documents, decks, code, apps and media, free to start. It is, as of this week and after a small argument with a text file, on the App Store too.