I have been cleaning up the public structure for a procurement workspace, and one thing became clearer than I expected: sitemap.xml, robots.txt, and llms.txt should not try to do the same job.
I used to think about this mostly as an indexing problem. Can search engines find the important pages? Are the routes listed? Is anything blocked by mistake?
That still matters, but it is only one layer.
The way I now split it:
- sitemap.xml is the page list. It should stay boring and stable.
- robots.txt is the crawl boundary. It tells crawlers what is allowed and what should be left alone.
- llms.txt is closer to a project note. It explains what the site is, which pages matter, and how not to misread the product.
For a procurement workspace, that last part matters. MapleBridge helps buyers create RFQs, invite their own supplier contacts and compare the quotes those suppliers submit. The project note should make that boundary clear: there is no supplied directory of verified factories or automatic introduction service.
I put the current plain-text version here:
https://maplebridge.io/llms.txt
I do not expect this file to magically create traffic. The practical value for me is simpler: it forces the product structure to be written down in a way that a person, a crawler, or a tool can inspect without guessing.
Top comments (0)