A practical guide for developers and vibe coders: why generated sites are often invisible to ChatGPT, Perplexity and Claude, how to plug a site check straight into your AI assistant via MCP, and which prompts make it find and fix the problems in code on its own. Plus numbers from checks of 41 sites.
Building a site in an evening with Cursor, Claude Code, Lovable or Bolt is the norm now. A landing page, a catalog, a services site – the assistant generates everything: layout, components, even the copy. The site looks good, loads fast, works.
Then it turns out that when someone asks ChatGPT "recommend a service for X", it recommends anyone but you. Perplexity doesn't cite you. Google AI Overviews quotes a competitor.
The reason is usually not the content but a few technical things the assistant didn't think about during generation, because you didn't ask it to. The good news: the same assistants are great at fixing them – if you give them an exact list of problems right in the editor. That's what this article is about.
Who we are and where the numbers come from
We build Zgadka – a service that audits sites for classic search and AI assistants. It checks whether the bots of ChatGPT, Perplexity, Claude, Gemini and Google can open and read a site, whether it's clear to them what the company is and what each page is about, whether there's anything in the text worth quoting, and what is known about the brand outside the site. Plus classic technical SEO: indexing, markup, speed.
The result is not just a score but a prioritized task list: what's wrong, why it matters, how many points a fix is worth, and ready-made code or a file for that specific site. There's also tracking of whether ChatGPT, Gemini and Google AI mention you in answers to customer questions, and analytics of visits coming from AI. And the MCP server this article is about delivers all of that straight into the AI assistant in your editor.
Where the data in this article comes from. All numbers below come from our own checks: we took the latest check of each site and counted the share of sites with each problem. Only aggregate percentages made it into the article – no URLs, names or any data about owners. The code and command examples are typical of what our audit finds and suggests fixing.
What breaks most often: the numbers
Here is aggregate data for 41 sites (the latest check of each; 46% are .ua domains), November 2026. The sample is small and doesn't claim to be market research: these are sites our users checked, plus competitors they added for comparison. But the patterns are telling.
The gap between SEO and AI readiness. The average classic SEO score is 81 out of 100; the average AI search readiness (GEO) score is 65. In other words, sites that are reasonably well built for Google are noticeably less prepared to be read and cited by assistants.
AI bots are cut off more often than you'd think:

Share of sites where an AI bot can't read the page: blue – disallowed in robots.txt, orange – refused by the server or CDN. 41 sites, November 2026.
| Bot | Disallowed in robots.txt | Blocked by server or CDN | Total |
|---|---|---|---|
OAI-SearchBot (ChatGPT search) |
12% | 10% | 22% |
GPTBot (OpenAI training) |
10% | 22% | 32% |
ClaudeBot |
12% | 15% | 27% |
PerplexityBot |
12% | 12% | 24% |
One in five sites is closed to ChatGPT search, and in about half of those cases it's not robots.txt but the CDN, bot protection or rate limiting. The owner usually has no idea.
Other findings:
- 15% of sites don't show their main text without JavaScript. Googlebot renders JS; most AI bots don't. To them the page is empty.
- 29% have no JSON-LD markup at all.
-
80% have no
Organizationmarkup withsameAs– the machine-readable "this site = this company = these profiles". - 88% of brands are not in Wikidata.
- 25% have no sitemap.
-
68% have no
llms.txt. To be fair, this is the least critical item: no major AI search engine has publicly confirmed using it. But it's useful for assistants in the IDE, and it takes a minute to make.
The most common platforms in the sample are WordPress and Next.js (12% each), followed by OpenCart and Nuxt.
Why generated sites are especially vulnerable
If you vibe-code, you almost certainly have one of these cases.
1. An SPA without server rendering. Lovable, Bolt and many "React + Vite" templates generate a single-page app by default: the server returns an empty <div id="root">, and all text is drawn by JavaScript in the browser. A human sees the site. The ChatGPT or Perplexity bot sees this:
<body>
<div id="root"></div>
<script type="module" src="/assets/index-a1b2c3.js"></script>
</body>
Check it in 5 seconds:
curl -s https://your-site.com/ | sed -e 's/<script[^>]*>.*<\/script>//g' -e 's/<[^>]*>/ /g' | wc -w
If you get a few dozen words while the page in the browser has hundreds, your site is empty as far as AI search is concerned.
2. The "boring" files are missing. The assistant generates what you asked for: pages and components. robots.txt, sitemap.xml, canonical URLs, meta descriptions, schema.org markup – these are what both humans and models usually forget.
3. Default deployment. A site on Vercel, Netlify or behind Cloudflare with default settings. Cloudflare has a separate toggle for blocking AI bots, and for new domains it may be on by default. robots.txt is perfect, and OAI-SearchBot gets a 403.
4. Copy "for looks". Generated copy often consists of nice generic phrases without a single number, price or specific detail. There's nothing for an assistant to quote.
All of this is fixable in code. The only question is how to give the assistant an exact list of what's wrong.
The idea: let the assistant check and fix it itself
Model Context Protocol (MCP) is the standard an AI assistant in your editor uses to connect external tools. We built an MCP server for Zgadka, a service that audits sites for search and AI assistants. Once connected, the assistant gets tools such as:
-
check_site– run a site check; -
get_report– scores and the list of problems; -
get_fix_task– a ready-made task for the agent: what to fix, in what order, with code and rules; -
get_ready_files– readyllms.txt,robots.txt, JSON-LD and FAQ built from your site's data; -
get_ai_answers– what ChatGPT, Gemini and Google AI currently answer to your customers' questions.
You open the site's repository, write one sentence – and the assistant checks the site, picks up the tasks, fixes the code and explains what's left to do by hand.
Why not just ask the assistant to check everything itself
A fair question: Claude Code or Cursor can already run curl and read HTML. Why a separate service?
Because the repository and the terminal show only part of the picture. The assistant sees your code and can fetch a page. But what decides whether you get cited lives elsewhere – and each source needs its own API, a key, and often payment:
| What you need to know | Where it comes from | Can the assistant get it alone |
|---|---|---|
| Whether the server and CDN let in each of 14 AI bots | requests with each bot's User-Agent, comparing responses | partly, by hand, one at a time |
| Speed for real visitors | Google PageSpeed Insights and Chrome UX Report data | no, needs a Google API key |
| Whether the brand is in Wikidata, Wikipedia, the Google panel | Wikidata, Wikipedia, Google results | no, Google results can't be fetched directly |
| Reviews and discussions of the brand on other sites | Google search for the brand | no |
| How many sites link to yours | backlink databases | no, these are paid databases |
| Whether the site is on Google's unsafe lists | Google Web Risk | no, needs a key and billing |
| Domain age, certificate expiry | domain registries (RDAP), TLS | partly |
| What ChatGPT, Gemini and Google AI answer to customer questions right now | each service's API with web search | no, needs keys and billing for each |
| How you compare with your niche | scores of other checked sites | no, it doesn't have that data |
| How many people actually come from AI | analytics of visits from ChatGPT, Perplexity, Gemini | no, not without an analytics snippet on the site |
Collecting all this yourself is doable, but it's a dozen sign-ups, keys and bills, plus code that merges the answers into one picture and sets priorities. Through MCP the assistant gets the merged result: one call, and it has the full list of problems from all these sources, sorted by what gives the most, with ready-made code.
The split of roles is simple: the service collects data that isn't in the repository, and the assistant does what it does best – changes code.
How to connect in 2 minutes
First you need a token: account → "MCP server" → "Create token". The token is shown once – save it right away. Then set up your tool.
Claude Code – one command in the terminal:
claude mcp add --transport http zgadka https://zgadka.com/mcp --header "Authorization: Bearer $ZGADKA_TOKEN"
Cursor – the file ~/.cursor/mcp.json (or .cursor/mcp.json in the project):
{
"mcpServers": {
"zgadka": {
"url": "https://zgadka.com/mcp",
"headers": { "Authorization": "Bearer zgk_your_token" }
}
}
}
Codex – ~/.codex/config.toml, with the token in an environment variable:
[mcp_servers.zgadka]
url = "https://zgadka.com/mcp"
bearer_token_env_var = "ZGADKA_TOKEN"
VS Code (GitHub Copilot, agent mode) – .vscode/mcp.json:
{
"servers": {
"zgadka": {
"type": "http",
"url": "https://zgadka.com/mcp",
"headers": { "Authorization": "Bearer zgk_your_token" }
}
}
}
Important: don't commit the file with the token to the repository. If you keep mcp.json in the project, add it to .gitignore or use an environment variable, as in the Codex example. If the token did end up in git, regenerate it in your account – the old one stops working immediately.
Prompts that work
Check and fix everything at once:
Check your-site.com in Zgadka and fix everything you find in this project, from critical tasks to minor ones. At the end, briefly describe the changes and what's left to do by hand.
In Claude Code there's a ready-made command for this – the fix_site prompt in the MCP server menu.
Just look, no changes:
Check your-site.com in Zgadka and explain the three most important problems in plain words. Don't change anything in the code.
One specific thing:
Get the ready files for your-site.com from Zgadka and add llms.txt and JSON-LD Organization to the project. Take company data only from the site; mark anything missing as TODO.
After deploying:
Check your-site.com in Zgadka again and compare the scores with the previous check.
What AI answers about your niche (with the full report):
Show what ChatGPT and Gemini answer to customer questions for your-site.com and whom they cite instead of us.
What the assistant usually changes
The task the assistant gets from get_fix_task contains not only the list of problems but also rules: fix minimally, don't touch the design or page URLs, don't invent facts about the company (phone numbers, prices, founding years – only from the site, mark the rest as TODO), and if the site's code isn't in the project, don't create it from scratch but write instructions instead. Each task has an estimate: how many points it adds and how long it takes.
Here's what most often changes in typical stacks.
Next.js (App Router) – the assistant adds the service files the framework can generate itself:
// app/robots.ts
import type { MetadataRoute } from "next";
export default function robots(): MetadataRoute.Robots {
return {
rules: [
{ userAgent: ["OAI-SearchBot", "ChatGPT-User", "PerplexityBot", "Claude-SearchBot"], allow: "/" },
{ userAgent: "*", allow: "/" },
],
sitemap: "https://your-site.com/sitemap.xml",
};
}
// app/sitemap.ts
import type { MetadataRoute } from "next";
export default function sitemap(): MetadataRoute.Sitemap {
return ["", "/pricing", "/about", "/faq"].map((p) => ({
url: `https://your-site.com${p}`,
lastModified: new Date(),
}));
}
and JSON-LD in the root layout:
// app/layout.tsx (excerpt)
const org = {
"@context": "https://schema.org",
"@type": "Organization",
name: "Company name",
url: "https://your-site.com",
logo: "https://your-site.com/logo.png",
sameAs: ["https://www.linkedin.com/company/…", "https://www.facebook.com/…"],
};
<script type="application/ld+json" dangerouslySetInnerHTML={{ __html: JSON.stringify(org) }} />
It also moves key text from client components ("use client") into server components so it ends up in the HTML.
React + Vite (SPA, including from Lovable or Bolt) – here the assistant can't "just add a file": the problem is the architecture. It usually proposes one of these options and explains the trade-offs:
- prerender key pages at build time (a Vite plugin or a separate CI step) – fastest;
- move to a framework with server rendering (Next.js, Remix, Astro) – reliable, but takes longer;
- at the very least –
robots.txt,sitemap.xmland JSON-LD as static files inpublic/andindex.html, so bots see at least the name, description and structure.
Any stack:
- rules for AI bots in
robots.txt(allow search bots; training bots are up to you); -
llms.txtat the root with a description of the site and its key pages; - unique
titleanddescriptionfor each page; - an FAQ block with short, direct answers to customer questions.
What the assistant can't do from code – and will honestly say so in the summary:
- turn off AI bot blocking in Cloudflare or another CDN (that's a dashboard setting);
- add the site to Google Search Console and Bing Webmaster Tools (ChatGPT and Copilot find sites via Bing);
- create a Google Business profile, Wikidata entries, directory listings, reviews.
What it looks like
Here's a typical session. The site and numbers are illustrative, but the format of the tool responses and the flow of work are real.

The assistant runs the check, takes prioritized tasks with ready files, changes six files and separately lists what's left to do by hand. Illustrative example: a made-up site, shop.example.
After deploying and doing the manual steps – a re-check with the same prompt:

Scores "before" and "after" by category. AI access grows the most – it depends only on code and settings. Reputation grows slowly: that's work outside the site. Illustrative example.
And so you don't have to write the prompt yourself, the full report includes a ready message for the assistant with that report's number:

The block in the full report: three connection steps and a message you just copy into the assistant.
What's free, what isn't – and security
- A site check is free at the basic level. The full report with all tasks and ready files is paid. The same rules and limits apply via MCP as on the website: the assistant gets no more than you see in the browser.
- Paid actions (opening a full report from your balance, an out-of-schedule tracking check) are flagged separately for the assistant, and the description says plainly: ask the user first. Modern clients also show a confirmation before such actions.
- The token is stored on the server only as a hash, shown in full once, and regenerating it disables the old one instantly.
- The assistant acts on behalf of your account and sees only your checks and projects.
A checklist for tonight
Even without any services, here's what to check on a generated site:
- [ ]
curlof the home page returns text, not an empty<div id="root">. - [ ]
OAI-SearchBot,PerplexityBot,Claude-SearchBotare not blocked in robots.txt or in the CDN. - [ ] There's a
sitemap.xmland it's listed in robots.txt. - [ ] The site is added to Google Search Console and Bing Webmaster Tools.
- [ ] There's JSON-LD
OrganizationwithsameAs; for products –Productwith price and currency. - [ ] Key customer questions are separate headings with the answer in the first sentence.
- [ ] The copy has specifics: prices, timelines, numbers, terms.
And if you don't feel like going through it by hand – ask the assistant to do it for you.
The numbers and examples in this article come from checks in Zgadka, where a basic site check is free; MCP setup is in your account, under "MCP server".
About the authors. We are the Zgadka team: developers building a technical site audit service for classic search and AI assistants. Every day we dig into why specific sites don't make it into the answers of ChatGPT, Perplexity, Gemini and Google, and turn that into tasks with ready-made code – for people in the browser and for AI assistants via MCP. We write about what we see in check data, without marketing promises. Questions, cases and disagreement – in the comments or at info@zgadka.com.
Top comments (0)