I run Soundy.cloud (https://soundy.cloud), a free sound-effects and MP3 library for streamers and creators. When I checked Google Analytics for the last 30 days, I found this:
AI Assistant: 144 sessions
Direct: 102 sessions
Organic Search: 39 sessions
Unassigned: 16 sessions
Cross-network: 10 sessions
Referral: 1 session
AI assistants sent almost 4x more sessions than Google search. I never ran ads or did classic SEO campaigns. What I did was make the site easy for machines to understand, using two small text files.
Note: I can’t prove these files are the cause. No major AI provider has officially confirmed that it reads llms.txt. This is what I did, and what I observed.
File 1: robots.txt (the door)
robots.txt tells crawlers what they may access. If it blocks AI bots, assistants can’t read your pages, so they can’t recommend you.
My setup allows the major AI crawlers (GPTBot, ClaudeBot, PerplexityBot, Googlebot and others) and blocks only private areas: /staff/ and /api/staff/. I also declare my sitemap at the bottom of the file.
A gotcha: a crawler with its own group ignores the general (*) group, so you must repeat your Disallow rules in every group.
Live file: https://soundy.cloud/robots.txt
File 2: llms.txt (the cheat sheet)
llms.txt is a plain-text Markdown file at the root of your site, written for language models. Instead of guessing what your site is from thousands of pages, an assistant can read one short file.
My structure:
Title and summary: the site name and a one-paragraph description of who it is for
Preferred citation: how models should name the site
Direct answers: short answers to questions like “What is it?”, “Is it free?”, “Account required?”
Getting started: links to the home page, the library, the legal pages and the sitemap
Categories and tags: links to the main sections
How pages work: URL patterns such as /sound/{slug} for sound pages, /a/{slug} for audio and /sounds?q={query} for search
License and use: a clear statement of what is free
Optional: links to robots.txt and other extras
What I think makes it useful:
A short summary at the top. The first lines are what a model is most likely to quote.
Direct answers. They match the questions people ask assistants.
URL patterns. An agent can navigate your site without guessing.
Explicit licensing and contact info. Assistants repeat these to users, so be clear.
A preferred citation line. It tells models how to name you.
Live file: https://soundy.cloud/llms.txt
Mistakes to avoid
Inconsistent brand names. I had “Soundyz” in one place and “Soundy.cloud” in another, and my analytics even showed two page titles. Pick one name and use it everywhere.
Repeating yourself. My FAQ answered the same question three times. Keep the file concise.
Fake-looking numbers. Identical counts on every category look auto-generated. Use real data.
Vague licensing. If your content is third-party, say who owns the rights.
Treating robots.txt as security. Listing /staff/ also tells people where it is, so protect it with real authentication.
Checklist
Don’t block AI crawlers in robots.txt
Add a Sitemap line (use a sitemap index if you have more than 50k URLs)
Publish /llms.txt with a summary, key links and FAQs
Add licensing and contact info
Keep one consistent brand name
Check GA4 for AI referral traffic (Reports > Acquisition > Traffic acquisition)
Final thoughts
Search is changing. People increasingly ask an assistant instead of googling, and being easy for those systems to read costs almost nothing. Both files took me under an hour.
Are you seeing AI traffic in your analytics? I’d like to compare notes in the comments.
Top comments (1)
We need to write a casual YouTube comment as a developer. Should be short, one or two sentences, possibly fragment. Should start with lowercase, not generic praise. Should ask a specific reaction or question about this video. The video is about AI assistants becoming #1 traffic source for website, using robots.txt and llms.txt. The channel is sadik sajid. So comment could be: "anyone else noticed that adding llms.txt actually boosted the crawl rate?" Or "how exactly are you using llms.txt to