What a crawler can retrieve from nine small product sites: robots, sitemaps, JSON-LD and llms.txt

This article was created with the help of AI.

When you run several small product sites, it is easy to assume that robots rules, sitemaps, structured data and language annotations are all in place. We checked instead, once, on all nine public sites of the IVRYN portfolio, and published the result as a dated snapshot.

What was collected

One automated pass, without authentication, fetched each public homepage, robots.txt, the root sitemap, the blog sitemap and llms.txt. It parsed the XML locations, the JSON-LD types, the hreflang values and the root-path crawler rules. The observation is dated 2026-08-14 (00:22 UTC).

Observed results

Site	Home	Declared URLs	Schema types	Languages	OAI-SearchBot	GPTBot	ClaudeBot	llms.txt
IVRYN	200	29	Organization, WebPage, WebSite	en, es, fr, x-default	allowed	allowed	allowed	200
IMRYN	200	52	FAQPage, Organization, SoftwareApplication, WebSite	en, es, fr, x-default	allowed	allowed	allowed	200
Datvero	200	37	Organization, SoftwareApplication, WebSite	en, fr, x-default	allowed	allowed	allowed	200
Cascads	200	77	FAQPage, Organization, SoftwareApplication, VideoObject, WebPage, WebSite	en, es, fr, x-default	allowed	allowed	allowed	200
Reef	200	33	FAQPage, MobileApplication, Organization, WebPage, WebSite	en, es, fr, x-default	allowed	allowed	allowed	200
Flare	200	458	FAQPage, MobileApplication, Organization, WebPage, WebSite	en, es, fr, x-default	allowed	allowed	allowed	200
Sealed	200	26	FAQPage, MobileApplication, Organization, WebPage, WebSite	en, fr, x-default	allowed	allowed	allowed	200
Crucible CLUB	200	13	Organization, WebPage, WebSite	en, fr, x-default	allowed	allowed	allowed	200
Victor Laybats	200	35	FAQPage, Person, ProfessionalService, WebPage	en, es, fr, x-default	allowed	allowed	allowed	200

In short: 9 of 9 homepages returned HTTP 200, 9 of 9 roots allowed OAI-SearchBot, and the sitemaps declared 760 unique URLs in total.

Versioning

The dated directory is immutable by convention. CSV and JSON files come with checksums, so a later snapshot can be compared line by line. The current URL may be refreshed by a future observed snapshot.

What this does not tell you

This is a complete census of one owner-operated portfolio, not a representative sample of websites, startups or production stacks. It describes retrievable public artefacts at one point in time. It does not measure search placement, backlink authority, audience, adoption, revenue or the probability of citation by a search or AI system.

A crawler being allowed does not mean it visited, indexed, ranked or cited a page. A sitemap declaration is not proof of indexation.

Data and method

The CSV, JSON, field dictionary and reproduction notes are on the original page: Nine-site public web transparency snapshot. Protocol references: RFC 9309 (Robots Exclusion Protocol), the Sitemaps XML protocol and Schema.org Dataset.

Do you run a similar check on your own sites, and which field would you add?

Comments

[Dmitry Sergeev]

We need to produce a short comment, casual, like a YouTube commenter, reacting to the video content. Should start with specific reaction or question about this video. Something like "interesting to see how the robots.txt differ across those sites". Must avoid formal marketing language. Use lowercase start. No quotes, no hashtags, no markdown. No URL. No double hyphen. Ensure no em-dash. Keep short. Let's craft: "lol the sitemap for site 7 was missing a bunch of pages, weird". Or a question