This is a submission for the Hacktoberfest Open-Source AI Challenge Week 4: Ghost in the Machine
What I Built
Most modern web developers test their site in Chrome. It loads cleanly, interactions are snappy, and the Lighthouse score looks great.
Then an autonomous AI agent or crawler (GPTBot, ClaudeBot, PerplexityBot) visits your site over raw HTTP. It does not wait for client-side JavaScript bundles to hydrate. It does not click your buttons or wait for async single-page app routers.
What does it see?
Nothing. Literally:
<div id="root"></div>
Your site becomes a "Ghost in the Machine"—a live, functioning web app that is completely invisible and inaudible to modern autonomous AI.
To fix this gap without requiring complex enterprise SEO setups, we launched SiteReadyFor.SI: a fast, technical AI-crawler readiness engine that audits your public pages and tells you exactly what bots see before you spend weeks wondering why search engines and answer engines ignore you.
What SiteReadyFor.SI Audits
Instead of guessing or relying on black-box ranking promises, SiteReadyFor.SI inspects the raw mechanics of bot discovery:
- SPA Ghost Detection: Analyzes your initial HTML payload to detect whether content exists on the wire or if it is trapped behind client-side execution.
-
AI Crawler Permissions: Tests exact policy matching for major autonomous agents (
GPTBot,ClaudeBot,PerplexityBot,Google-Extended, andChatGPT-User). - Structured Data Validation: Checks JSON-LD entity structures so machines understand your product, author, or organization without hallucination.
-
Instant
llms.txtDrafts: Automatically generates a clean, markdown-friendlyllms.txtfile tailored to your audited pages so autonomous tools can ingest your core information effortlessly.
Live Demo & Free Audit
You do not need an account or API keys to test your site.
👉 Run a free single-page check directly at sitereadyfor.si
Enter your homepage or documentation URL, and within seconds you get an unvarnished audit:
- Does your
robots.txtsilently wall off AI crawlers? - Is your initial HTML sparse or fully parseable?
- Are your structured schemas valid?
- Download a ready-to-deploy
llms.txtfile immediately.
Why Open Innovation Matters
The modern web is at a crossroads between open access and closed walled gardens:
- Transparency Over Black Boxes: Many SEO platforms sell expensive "AI visibility" metrics based on proprietary black-box guesses. We believe technical readiness should be open, deterministic, and verifiable by the developer who built the site.
-
Open Standards vs. Platform Lock-in: Formats like
llms.txtand open JSON-LD schemas give developers control over their own content distribution, preventing third-party search monopolies from acting as gatekeepers. - Privacy First: Validating technical signals should never require sending your site's private visitor data or credentials to third-party ad networks.
The Spirit of It
For this Hacktoberfest theme (Ghost in the Machine), we wanted to highlight an unspoken reality of the modern web: thousands of stunning web applications are ghosts to the very AI systems meant to index and understand them.
Before investing in marketing or citation tracking, ensure your foundation is not a blank page.
Test your web app right now at SiteReadyFor.SI.
Top comments (14)
The llms.txt generation is a smart feature. Is there a standard format for that file yet, or are you defining one as you go? I'd love to see an example of what the output looks like.
Yes, exactly! It adheres to the emerging llms.txt specification originally proposed by Jeremy Howard (Answer.ai / fast.ai).
Rather than inventing a proprietary schema, the idea is to provide an intentional, clean Markdown map that LLM agents and web crawlers can ingest directly without burning context tokens on navigation boilerplate or getting trapped in client-side routing.
Here is a quick example of what SiteReadyFor.SI outputs after parsing a site's discovered routes:
The llms.txt approach is smart. Instead of burning tokens on navigation boilerplate, the agent gets a clean Markdown map of the site. It's the difference between handing someone a sitemap and making them click through every page.
That analogy is brilliant "handing someone a sitemap vs making them click through every page" captures the entire philosophy perfectly.
When an autonomous agent has to crawl 5 raw HTML pages just to answer a simple question, it burns thousands of context tokens on header menus, cookie consent banners, and footer links before it even reaches the actual documentation. With
llms.txt, you compress that entire discovery cycle into a clean, predictable map.Curious have you started experimenting with
llms.txton any of your own projects yet, or are you guys still relying primarily on standard XML sitemaps?Honestly, I'm still mostly relying on standard XML sitemaps for the projects I maintain. The llms.txt idea is compelling, but I haven't had a project where the agent-discovery problem was urgent enough to justify the switch. That said, the more I see examples like this, the more I'm tempted to try it.
That’s completely fair, if traditional search is already driving your traffic, standard XML sitemaps do the heavy lifting just fine, and nobody needs another maintenance chore.
The neat part is that it’s actually not an "either/or" switch!
sitemap.xmlstays strictly in place for Google & Bing, whilellms.txtjust sits right next to it in your root directory as a lightweight companion for LLM crawlers.If you ever feel like testing the waters without committing sprint time, documentation or content-heavy sites are usually the lowest-friction place to start: you can literally drop a static 15-line Markdown file into your
/publicfolder in two minutes. Anyone pointing Cursor, Claude, or Perplexity at your domain to reference your site gets clean, hallucination-free context instantly.Whenever you decide to give it a spin on a side project, would love to hear your thoughts on the workflow!
That makes total sense. Framing it as a lightweight companion rather than a replacement lowers the barrier to entry significantly. I'll definitely try dropping a 15-line file into my /public folder this weekend!
浏览器里跑得顺不代表机器能读到内容,只看原始 HTTP 响应这一步很关键,很多 SPA 在这里就是一个空 div。把 robots.txt 对各家爬虫的精确匹配单独列出来也很有用,光写允许全部爬取很容易漏掉某个 agent。下一步我会更在意首屏 HTML 里是否已经包含核心内容,而不是等 JS 注水后再补。
Spot on! You nailed the exact architectural problem we wanted to highlight.
Most developers forget that autonomous crawlers operate on strict latency and compute budgets they aren't spinning up heavy headless Chromium instances with full V8 runtimes just to scrape text. If the raw HTTP payload doesn't have the content in the first 20KB, to a machine, it simply doesn't exist.
And regarding
robots.txt: exactly. A lot of teams assumeUser-agent: *covers everything, completely unaware that bots likeChatGPT-UservsGPTBotorGoogle-Extendedhave specific priority rules that silently bypass or trigger blocks.Appreciate the thoughtful feedback! That focus on initial-payload content density rather than post-hydration backfilling is definitely the way forward for modern technical SEO.
Just curious what tech stack are you guys primarily running right now? Are you leaning more toward full SSR (Next.js / Remix) or static generation (SSG) to bridge that initial payload gap?
Some comments may only be visible to logged-in visitors. Sign in to view all comments.