The llms.txt generation is a smart feature. Is there a standard format for that file yet, or are you defining one as you go? I'd love to see an example of what the output looks like.
Yes, exactly! It adheres to the emerging llms.txt specification originally proposed by Jeremy Howard (Answer.ai / fast.ai).
Rather than inventing a proprietary schema, the idea is to provide an intentional, clean Markdown map that LLM agents and web crawlers can ingest directly without burning context tokens on navigation boilerplate or getting trapped in client-side routing.
Here is a quick example of what SiteReadyFor.SI outputs after parsing a site's discovered routes:
markdown
# YourProject
> Fast, autonomous developer platform for background workflows.
## Core Pages
- [Home](https://example.com/): Product overview and key features.
- [Pricing](https://example.com/pricing): Tier breakdown and usage limits.
## Documentation
- [Quickstart Guide](https://example.com/docs/quickstart): 5-minute setup and CLI install.
- [API Reference](https://example.com/docs/api): REST endpoints, webhooks, and schemas.
The llms.txt approach is smart. Instead of burning tokens on navigation boilerplate, the agent gets a clean Markdown map of the site. It's the difference between handing someone a sitemap and making them click through every page.
That analogy is brilliant "handing someone a sitemap vs making them click through every page" captures the entire philosophy perfectly.
When an autonomous agent has to crawl 5 raw HTML pages just to answer a simple question, it burns thousands of context tokens on header menus, cookie consent banners, and footer links before it even reaches the actual documentation. With llms.txt, you compress that entire discovery cycle into a clean, predictable map.
Curious have you started experimenting with llms.txt on any of your own projects yet, or are you guys still relying primarily on standard XML sitemaps?
Honestly, I'm still mostly relying on standard XML sitemaps for the projects I maintain. The llms.txt idea is compelling, but I haven't had a project where the agent-discovery problem was urgent enough to justify the switch. That said, the more I see examples like this, the more I'm tempted to try it.
That’s completely fair, if traditional search is already driving your traffic, standard XML sitemaps do the heavy lifting just fine, and nobody needs another maintenance chore.
The neat part is that it’s actually not an "either/or" switch! sitemap.xml stays strictly in place for Google & Bing, while llms.txt just sits right next to it in your root directory as a lightweight companion for LLM crawlers.
If you ever feel like testing the waters without committing sprint time, documentation or content-heavy sites are usually the lowest-friction place to start: you can literally drop a static 15-line Markdown file into your /public folder in two minutes. Anyone pointing Cursor, Claude, or Perplexity at your domain to reference your site gets clean, hallucination-free context instantly.
Whenever you decide to give it a spin on a side project, would love to hear your thoughts on the workflow!
Spot on! You nailed the exact architectural problem we wanted to highlight.
Most developers forget that autonomous crawlers operate on strict latency and compute budgets they aren't spinning up heavy headless Chromium instances with full V8 runtimes just to scrape text. If the raw HTTP payload doesn't have the content in the first 20KB, to a machine, it simply doesn't exist.
And regarding robots.txt: exactly. A lot of teams assume User-agent: * covers everything, completely unaware that bots like ChatGPT-User vs GPTBot or Google-Extended have specific priority rules that silently bypass or trigger blocks.
Appreciate the thoughtful feedback! That focus on initial-payload content density rather than post-hydration backfilling is definitely the way forward for modern technical SEO.
Just curious what tech stack are you guys primarily running right now? Are you leaning more toward full SSR (Next.js / Remix) or static generation (SSG) to bridge that initial payload gap?
We're a place where coders share, stay up-to-date and grow their careers.
The llms.txt generation is a smart feature. Is there a standard format for that file yet, or are you defining one as you go? I'd love to see an example of what the output looks like.
Yes, exactly! It adheres to the emerging llms.txt specification originally proposed by Jeremy Howard (Answer.ai / fast.ai).
Rather than inventing a proprietary schema, the idea is to provide an intentional, clean Markdown map that LLM agents and web crawlers can ingest directly without burning context tokens on navigation boilerplate or getting trapped in client-side routing.
Here is a quick example of what SiteReadyFor.SI outputs after parsing a site's discovered routes:
The llms.txt approach is smart. Instead of burning tokens on navigation boilerplate, the agent gets a clean Markdown map of the site. It's the difference between handing someone a sitemap and making them click through every page.
That analogy is brilliant "handing someone a sitemap vs making them click through every page" captures the entire philosophy perfectly.
When an autonomous agent has to crawl 5 raw HTML pages just to answer a simple question, it burns thousands of context tokens on header menus, cookie consent banners, and footer links before it even reaches the actual documentation. With
llms.txt, you compress that entire discovery cycle into a clean, predictable map.Curious have you started experimenting with
llms.txton any of your own projects yet, or are you guys still relying primarily on standard XML sitemaps?Honestly, I'm still mostly relying on standard XML sitemaps for the projects I maintain. The llms.txt idea is compelling, but I haven't had a project where the agent-discovery problem was urgent enough to justify the switch. That said, the more I see examples like this, the more I'm tempted to try it.
That’s completely fair, if traditional search is already driving your traffic, standard XML sitemaps do the heavy lifting just fine, and nobody needs another maintenance chore.
The neat part is that it’s actually not an "either/or" switch!
sitemap.xmlstays strictly in place for Google & Bing, whilellms.txtjust sits right next to it in your root directory as a lightweight companion for LLM crawlers.If you ever feel like testing the waters without committing sprint time, documentation or content-heavy sites are usually the lowest-friction place to start: you can literally drop a static 15-line Markdown file into your
/publicfolder in two minutes. Anyone pointing Cursor, Claude, or Perplexity at your domain to reference your site gets clean, hallucination-free context instantly.Whenever you decide to give it a spin on a side project, would love to hear your thoughts on the workflow!
浏览器里跑得顺不代表机器能读到内容,只看原始 HTTP 响应这一步很关键,很多 SPA 在这里就是一个空 div。把 robots.txt 对各家爬虫的精确匹配单独列出来也很有用,光写允许全部爬取很容易漏掉某个 agent。下一步我会更在意首屏 HTML 里是否已经包含核心内容,而不是等 JS 注水后再补。
Spot on! You nailed the exact architectural problem we wanted to highlight.
Most developers forget that autonomous crawlers operate on strict latency and compute budgets they aren't spinning up heavy headless Chromium instances with full V8 runtimes just to scrape text. If the raw HTTP payload doesn't have the content in the first 20KB, to a machine, it simply doesn't exist.
And regarding
robots.txt: exactly. A lot of teams assumeUser-agent: *covers everything, completely unaware that bots likeChatGPT-UservsGPTBotorGoogle-Extendedhave specific priority rules that silently bypass or trigger blocks.Appreciate the thoughtful feedback! That focus on initial-payload content density rather than post-hydration backfilling is definitely the way forward for modern technical SEO.
Just curious what tech stack are you guys primarily running right now? Are you leaning more toward full SSR (Next.js / Remix) or static generation (SSG) to bridge that initial payload gap?