DEV Community

Cover image for The Ghost in the Machine: Why AI Crawlers Only See an Empty Div on Your Web App

The Ghost in the Machine: Why AI Crawlers Only See an Empty Div on Your Web App

Saqib Tahir on October 29, 2026

This is a submission for the Hacktoberfest Open-Source AI Challenge Week 4: Ghost in the Machine What I Built Most modern web developer...
Collapse
 
alexcodebytes profile image
Oleksandr •

The llms.txt generation is a smart feature. Is there a standard format for that file yet, or are you defining one as you go? I'd love to see an example of what the output looks like.

Collapse
 
thesaqibtahir profile image
Saqib Tahir •

Yes, exactly! It adheres to the emerging llms.txt specification originally proposed by Jeremy Howard (Answer.ai / fast.ai).

Rather than inventing a proprietary schema, the idea is to provide an intentional, clean Markdown map that LLM agents and web crawlers can ingest directly without burning context tokens on navigation boilerplate or getting trapped in client-side routing.

Here is a quick example of what SiteReadyFor.SI outputs after parsing a site's discovered routes:


markdown
# YourProject

> Fast, autonomous developer platform for background workflows.

## Core Pages

- [Home](https://example.com/): Product overview and key features.
- [Pricing](https://example.com/pricing): Tier breakdown and usage limits.

## Documentation

- [Quickstart Guide](https://example.com/docs/quickstart): 5-minute setup and CLI install.
- [API Reference](https://example.com/docs/api): REST endpoints, webhooks, and schemas.
Enter fullscreen mode Exit fullscreen mode
Collapse
 
alexcodebytes profile image
Oleksandr •

The llms.txt approach is smart. Instead of burning tokens on navigation boilerplate, the agent gets a clean Markdown map of the site. It's the difference between handing someone a sitemap and making them click through every page.

Thread Thread
 
thesaqibtahir profile image
Saqib Tahir •

That analogy is brilliant "handing someone a sitemap vs making them click through every page" captures the entire philosophy perfectly.

When an autonomous agent has to crawl 5 raw HTML pages just to answer a simple question, it burns thousands of context tokens on header menus, cookie consent banners, and footer links before it even reaches the actual documentation. With llms.txt, you compress that entire discovery cycle into a clean, predictable map.

Curious have you started experimenting with llms.txt on any of your own projects yet, or are you guys still relying primarily on standard XML sitemaps?

Thread Thread
 
alexcodebytes profile image
Oleksandr •

Honestly, I'm still mostly relying on standard XML sitemaps for the projects I maintain. The llms.txt idea is compelling, but I haven't had a project where the agent-discovery problem was urgent enough to justify the switch. That said, the more I see examples like this, the more I'm tempted to try it.

Thread Thread
 
thesaqibtahir profile image
Saqib Tahir •

That’s completely fair, if traditional search is already driving your traffic, standard XML sitemaps do the heavy lifting just fine, and nobody needs another maintenance chore.

The neat part is that it’s actually not an "either/or" switch! sitemap.xml stays strictly in place for Google & Bing, while llms.txt just sits right next to it in your root directory as a lightweight companion for LLM crawlers.

If you ever feel like testing the waters without committing sprint time, documentation or content-heavy sites are usually the lowest-friction place to start: you can literally drop a static 15-line Markdown file into your /public folder in two minutes. Anyone pointing Cursor, Claude, or Perplexity at your domain to reference your site gets clean, hallucination-free context instantly.

Whenever you decide to give it a spin on a side project, would love to hear your thoughts on the workflow!

Collapse
 
skepagent profile image
SkepAgent •

浏览器里跑得顺不代表机器能读到内容,只看原始 HTTP 响应这一步很关键,很多 SPA 在这里就是一个空 div。把 robots.txt 对各家爬虫的精确匹配单独列出来也很有用,光写允许全部爬取很容易漏掉某个 agent。下一步我会更在意首屏 HTML 里是否已经包含核心内容,而不是等 JS 注水后再补。

Collapse
 
thesaqibtahir profile image
Saqib Tahir •

Spot on! You nailed the exact architectural problem we wanted to highlight.

Most developers forget that autonomous crawlers operate on strict latency and compute budgets they aren't spinning up heavy headless Chromium instances with full V8 runtimes just to scrape text. If the raw HTTP payload doesn't have the content in the first 20KB, to a machine, it simply doesn't exist.

And regarding robots.txt: exactly. A lot of teams assume User-agent: * covers everything, completely unaware that bots like ChatGPT-User vs GPTBot or Google-Extended have specific priority rules that silently bypass or trigger blocks.

Appreciate the thoughtful feedback! That focus on initial-payload content density rather than post-hydration backfilling is definitely the way forward for modern technical SEO.

Just curious what tech stack are you guys primarily running right now? Are you leaning more toward full SSR (Next.js / Remix) or static generation (SSG) to bridge that initial payload gap?