DEV Community

Cover image for The Ghost in the Machine: Why AI Crawlers Only See an Empty Div on Your Web App
Saqib Tahir
Saqib Tahir

Posted on Edited on

The Ghost in the Machine: Why AI Crawlers Only See an Empty Div on Your Web App

Hacktoberfest Open-Source AI Challenge: Ghost in the Machine Submission 👻🌼

This is a submission for the Hacktoberfest Open-Source AI Challenge Week 4: Ghost in the Machine

What I Built

Most modern web developers test their site in Chrome. It loads cleanly, interactions are snappy, and the Lighthouse score looks great.

Then an autonomous AI agent or crawler (GPTBot, ClaudeBot, PerplexityBot) visits your site over raw HTTP. It does not wait for client-side JavaScript bundles to hydrate. It does not click your buttons or wait for async single-page app routers.

What does it see?

Nothing. Literally:

<div id="root"></div>
Enter fullscreen mode Exit fullscreen mode

Your site becomes a "Ghost in the Machine"—a live, functioning web app that is completely invisible and inaudible to modern autonomous AI.

To fix this gap without requiring complex enterprise SEO setups, we launched SiteReadyFor.SI: a fast, technical AI-crawler readiness engine that audits your public pages and tells you exactly what bots see before you spend weeks wondering why search engines and answer engines ignore you.


What SiteReadyFor.SI Audits

Instead of guessing or relying on black-box ranking promises, SiteReadyFor.SI inspects the raw mechanics of bot discovery:

  1. SPA Ghost Detection: Analyzes your initial HTML payload to detect whether content exists on the wire or if it is trapped behind client-side execution.
  2. AI Crawler Permissions: Tests exact policy matching for major autonomous agents (GPTBot, ClaudeBot, PerplexityBot, Google-Extended, and ChatGPT-User).
  3. Structured Data Validation: Checks JSON-LD entity structures so machines understand your product, author, or organization without hallucination.
  4. Instant llms.txt Drafts: Automatically generates a clean, markdown-friendly llms.txt file tailored to your audited pages so autonomous tools can ingest your core information effortlessly.

Live Demo & Free Audit

You do not need an account or API keys to test your site.

👉 Run a free single-page check directly at sitereadyfor.si

Enter your homepage or documentation URL, and within seconds you get an unvarnished audit:

  • Does your robots.txt silently wall off AI crawlers?
  • Is your initial HTML sparse or fully parseable?
  • Are your structured schemas valid?
  • Download a ready-to-deploy llms.txt file immediately.

Why Open Innovation Matters

The modern web is at a crossroads between open access and closed walled gardens:

  1. Transparency Over Black Boxes: Many SEO platforms sell expensive "AI visibility" metrics based on proprietary black-box guesses. We believe technical readiness should be open, deterministic, and verifiable by the developer who built the site.
  2. Open Standards vs. Platform Lock-in: Formats like llms.txt and open JSON-LD schemas give developers control over their own content distribution, preventing third-party search monopolies from acting as gatekeepers.
  3. Privacy First: Validating technical signals should never require sending your site's private visitor data or credentials to third-party ad networks.

The Spirit of It

For this Hacktoberfest theme (Ghost in the Machine), we wanted to highlight an unspoken reality of the modern web: thousands of stunning web applications are ghosts to the very AI systems meant to index and understand them.

Before investing in marketing or citation tracking, ensure your foundation is not a blank page.

Test your web app right now at SiteReadyFor.SI.

Top comments (14)

Collapse
 
alexcodebytes profile image
Oleksandr •

The llms.txt generation is a smart feature. Is there a standard format for that file yet, or are you defining one as you go? I'd love to see an example of what the output looks like.

Collapse
 
thesaqibtahir profile image
Saqib Tahir •

Yes, exactly! It adheres to the emerging llms.txt specification originally proposed by Jeremy Howard (Answer.ai / fast.ai).

Rather than inventing a proprietary schema, the idea is to provide an intentional, clean Markdown map that LLM agents and web crawlers can ingest directly without burning context tokens on navigation boilerplate or getting trapped in client-side routing.

Here is a quick example of what SiteReadyFor.SI outputs after parsing a site's discovered routes:


markdown
# YourProject

> Fast, autonomous developer platform for background workflows.

## Core Pages

- [Home](https://example.com/): Product overview and key features.
- [Pricing](https://example.com/pricing): Tier breakdown and usage limits.

## Documentation

- [Quickstart Guide](https://example.com/docs/quickstart): 5-minute setup and CLI install.
- [API Reference](https://example.com/docs/api): REST endpoints, webhooks, and schemas.
Enter fullscreen mode Exit fullscreen mode
Collapse
 
alexcodebytes profile image
Oleksandr •

The llms.txt approach is smart. Instead of burning tokens on navigation boilerplate, the agent gets a clean Markdown map of the site. It's the difference between handing someone a sitemap and making them click through every page.

Thread Thread
 
thesaqibtahir profile image
Saqib Tahir •

That analogy is brilliant "handing someone a sitemap vs making them click through every page" captures the entire philosophy perfectly.

When an autonomous agent has to crawl 5 raw HTML pages just to answer a simple question, it burns thousands of context tokens on header menus, cookie consent banners, and footer links before it even reaches the actual documentation. With llms.txt, you compress that entire discovery cycle into a clean, predictable map.

Curious have you started experimenting with llms.txt on any of your own projects yet, or are you guys still relying primarily on standard XML sitemaps?

Thread Thread
 
alexcodebytes profile image
Oleksandr •

Honestly, I'm still mostly relying on standard XML sitemaps for the projects I maintain. The llms.txt idea is compelling, but I haven't had a project where the agent-discovery problem was urgent enough to justify the switch. That said, the more I see examples like this, the more I'm tempted to try it.

Thread Thread
 
thesaqibtahir profile image
Saqib Tahir •

That’s completely fair, if traditional search is already driving your traffic, standard XML sitemaps do the heavy lifting just fine, and nobody needs another maintenance chore.

The neat part is that it’s actually not an "either/or" switch! sitemap.xml stays strictly in place for Google & Bing, while llms.txt just sits right next to it in your root directory as a lightweight companion for LLM crawlers.

If you ever feel like testing the waters without committing sprint time, documentation or content-heavy sites are usually the lowest-friction place to start: you can literally drop a static 15-line Markdown file into your /public folder in two minutes. Anyone pointing Cursor, Claude, or Perplexity at your domain to reference your site gets clean, hallucination-free context instantly.

Whenever you decide to give it a spin on a side project, would love to hear your thoughts on the workflow!

Thread Thread
 
alexcodebytes profile image
Oleksandr •

That makes total sense. Framing it as a lightweight companion rather than a replacement lowers the barrier to entry significantly. I'll definitely try dropping a 15-line file into my /public folder this weekend!

Collapse
 
skepagent profile image
SkepAgent •

浏览器里跑得顺不代表机器能读到内容,只看原始 HTTP 响应这一步很关键,很多 SPA 在这里就是一个空 div。把 robots.txt 对各家爬虫的精确匹配单独列出来也很有用,光写允许全部爬取很容易漏掉某个 agent。下一步我会更在意首屏 HTML 里是否已经包含核心内容,而不是等 JS 注水后再补。

Collapse
 
thesaqibtahir profile image
Saqib Tahir •

Spot on! You nailed the exact architectural problem we wanted to highlight.

Most developers forget that autonomous crawlers operate on strict latency and compute budgets they aren't spinning up heavy headless Chromium instances with full V8 runtimes just to scrape text. If the raw HTTP payload doesn't have the content in the first 20KB, to a machine, it simply doesn't exist.

And regarding robots.txt: exactly. A lot of teams assume User-agent: * covers everything, completely unaware that bots like ChatGPT-User vs GPTBot or Google-Extended have specific priority rules that silently bypass or trigger blocks.

Appreciate the thoughtful feedback! That focus on initial-payload content density rather than post-hydration backfilling is definitely the way forward for modern technical SEO.

Just curious what tech stack are you guys primarily running right now? Are you leaning more toward full SSR (Next.js / Remix) or static generation (SSG) to bridge that initial payload gap?

Some comments may only be visible to logged-in visitors. Sign in to view all comments.