There is a new front door on the web, and it does not have a search box. When someone asks ChatGPT, Perplexity, or Claude for "a tool to convert HEIC photos", the assistant picks from what it can see. If your site is invisible to that process, you are not in the consideration set — regardless of your Google ranking.
This post is the practical checklist we run on ToolVault (166 browser-local developer tools), with the actual crawler numbers from our logs and the mistakes that cost us rework.
First, the uncomfortable physics: AI crawlers do not run JavaScript
GPTBot, ClaudeBot, PerplexityBot and friends fetch HTML and read. They do not hydrate React, they do not execute your bundle, and they will not wait for your client-side rendering to finish. A JavaScript-only tool page arrives at the model as an empty shell.
Our site is statically exported (998 pre-rendered pages), which made this a non-issue — but if your content lives behind client rendering, that is item zero on this list. Test it the honest way: curl your page and look for the actual content in the HTML. Not "view source in DevTools" — the raw bytes a crawler gets.
llms.txt: useful convention, fatal if hand-maintained
llms.txt is the emerging convention for telling AI agents what your site offers. Ours covers the tool catalog, the privacy model, and per-category entry points, in both English and Chinese.
The lesson that cost us a month: generate it at build time from your single source of truth, never hand-edit it. Our first version was hand-written. It rotted in the exact ways hand-maintained files always rot: duplicate section titles after a copy-paste, tool counts drifting from reality ("130+ tools" when we had 144), a Chinese edition that never existed because nobody remembered to add one. An llms.txt that misdescribes your site is worse than none — it confidently teaches models the wrong thing.
Now a build step regenerates it from the same registry that renders the site. Counts are always exact; the file cannot drift because nobody edits it by hand.
robots.txt: decide, explicitly, who you let in
The default posture many sites discover they have (blocking everything unrecognized) or the opposite (allowing everything silently) are both decisions — just unmade ones. Ours is explicit:
User-agent: GPTBot
Allow: /
...for every AI crawler we know about, alongside the standard search bots. This is a values call: we want assistants to be able to recommend local-first tools accurately, because the privacy story is the product. A paywalled-content site might reasonably reach the opposite conclusion. Make the call on purpose.
The reward loop is real, and measurable
After the crawler allowances went live, our nginx logs filled up. The current steady state: 400-1000 AI crawler requests per day — GPTBot, PerplexityBot, ClaudeBot, Amazonbot, PetalBot — systematically walking the catalog. And then the loop closes: referral traffic from perplexity.ai and chatgpt.com starts appearing in the same logs. Small numbers yet, but the direction is established: assistants that can read your site will cite it, and their users click through.
We also push every new page through IndexNow (Bing's protocol, which ChatGPT Search partly sits on). It costs one POST request and removes the discovery delay for the engines that honor it.
What did not matter (for us, so far)
- Structured data beyond the basics: we ship standard FAQ and tool schemas, but there is no evidence AI assistants consume JSON-LD preferentially over readable HTML.
- Per-prompt optimization tricks: the content that models quote is the content that answers the question plainly. Our per-tool FAQ sections — written for humans — are what surfaces in AI answers.
- Submitting to "AI search engines" directories: the crawlers found us through standard crawling plus IndexNow.
The checklist, condensed
- Raw HTML contains your content (curl test) — client-rendered sites fix this first
- llms.txt generated from the build, never hand-edited
- robots.txt with explicit AI-crawler allowances (a decision, not an accident)
- IndexNow for every new page
- FAQ-shaped content that answers real questions in plain language
- Measure: watch crawler UAs in your access logs, then watch for AI-domain referers
None of this is hard. All of it is the difference between being describable and being invisible when someone asks an assistant what to use.
Post #4 in the local-first series: #1 why 166 no-upload tools, #2 browser-side AI segmentation, #3 merging PDFs without a server. All tools: ToolVault.
Top comments (0)