This article was originally published in Japanese on Zenn.
Introduction
Hi, I'm Kohei Matsunobe, a second-year student at ZEN University studying Intelligent Informatics (online). I build AI agents and apps as personal projects, including an iOS app called Kamimegu and apps built with Expo / React Native.
I recently wrote this on LinkedIn:
The skills web engineers need from now on are not only "building sites that are easy for humans to read." Building sites that AI agents (such as GrokBot and Muse) can correctly recognize is becoming essential.
When you build AI agents, you notice that an agent is also a reader of the web. We are moving from a world where people click links in search results to one where agents collect and interpret information on the web and present answers or actions. In that world, the question for site builders is not only how a page looks, but whether a machine can read its structure without getting it wrong. I think of this as DX for AI: Data Experience.
I call this viewpoint AIO (AI Optimization). In this article I turn the four points from my post into minimal code:
- Semantic HTML
- Structured data (JSON-LD)
- Access control with AI crawlers in mind (robots.txt), and llms.txt
- Easy-to-fetch data, APIs, and performance
A caveat up front: this is not a "do this and AI will definitely pick you up" guide. I haven't verified from primary sources how any specific AI agent parses a site, so I won't claim anything about that. What follows is based on official documentation and specs: how to make pages machine-readable, and how to signal your intent to crawlers.
1. Semantic HTML: let the tags carry the meaning
The problem
Two pages can look identical, yet a page made only of divs doesn't tell a machine which part is the heading, the main content, or the navigation. A human can tell by looking. Something reading the HTML has far fewer clues beyond the tags.
Before
<div class="header">
<div class="logo">My Blog</div>
<div class="menu">
<div><a href="/">Home</a></div>
<div><a href="/posts">Posts</a></div>
</div>
</div>
<div class="content">
<div class="title">What is AIO?</div>
<div class="date">2026-11-18</div>
<div class="text">Body text...</div>
</div>
<div class="footer">© 2026</div>
After
<header>
<p class="logo">My Blog</p>
<nav aria-label="Main">
<ul>
<li><a href="/">Home</a></li>
<li><a href="/posts">Posts</a></li>
</ul>
</nav>
</header>
<main>
<article>
<h1>What is AIO?</h1>
<p><time datetime="2026-11-18">November 18, 2026</time></p>
<p>Body text...</p>
</article>
</main>
<footer>© 2026</footer>
Key points:
- Put the page's primary content in
<main>(one per page). - Use
<article>for self-contained pieces and<nav>for navigation. - Use headings in order (
h1→h2→h3) without skipping levels. Don't pick a heading tag for its size; use CSS for that. - Use
<time datetime="...">for dates and meaningfulalttext for images.
This is also standard accessibility practice (screen readers). Here, "easy for humans to use" and "easy for machines to read" turn out to be the same work.
References: MDN: Semantics, MDN: <main>
Side note: if your content only appears in the HTML after JavaScript runs, clients that don't execute JavaScript will see nothing. Whether a client runs JavaScript is up to that client, so it's safer to include important content in the initial HTML response (SSR / SSG). If you export Expo or React Native code to the web, keep this in mind for the pages you want to be read, such as a product page.
2. Structured data (JSON-LD): add meaning in a machine-readable form
Semantic HTML describes a document's skeleton. Structured data states what a page is about (an article, an app, who wrote it) using the Schema.org vocabulary. Google recommends JSON-LD as the format (Google: Intro to structured data).
An article page (Article)
<script type="application/ld+json">
{
"@context": "https://schema.org",
"@type": "Article",
"headline": "What is AIO?",
"datePublished": "2026-11-18T09:00:00+09:00",
"dateModified": "2026-11-18T09:00:00+09:00",
"author": [
{
"@type": "Person",
"name": "Kohei Matsunobe",
"url": "https://zenn.dev/koheip"
}
],
"image": ["https://example.com/images/aio-cover.png"]
}
</script>
Google's Article documentation says there are no required properties; you add the recommended ones that apply (Google: Article structured data).
An app landing page (SoftwareApplication)
For an app's landing page, SoftwareApplication fits. These are dummy values:
<script type="application/ld+json">
{
"@context": "https://schema.org",
"@type": "SoftwareApplication",
"name": "Sample App",
"operatingSystem": "iOS",
"applicationCategory": "LifestyleApplication",
"offers": {
"@type": "Offer",
"price": "0",
"priceCurrency": "JPY"
}
}
</script>
Two notes:
- Keep the values consistent with what the page actually says. Don't put ratings or prices only in structured data when they don't appear on the page.
- If you want Google's software app rich result, there are required properties of its own:
name,offers.price, andaggregateRatingorreview(Google: Software app). If no real ratings or reviews exist, don't invent them. As a Schema.org vocabulary, the minimal form above is still valid.
You can browse the available types and properties at Schema.org/Article and Schema.org/SoftwareApplication. Adding structured data doesn't guarantee that any particular AI agent will use it. For Google Search features, the docs above are the primary source.
3. Crawler-aware access control: robots.txt and llms.txt
robots.txt: signal your intent per crawler
What matters here is that crawlers have different purposes. The vendors' own docs draw these distinctions:
| User agent / token | Vendor | Purpose as stated in the docs |
|---|---|---|
GPTBot |
OpenAI | Crawling content that may be used to train generative AI foundation models |
OAI-SearchBot |
OpenAI | Surfacing sites in ChatGPT's search features |
ChatGPT-User |
OpenAI | User-initiated actions. Not automatic crawling; the docs say robots.txt rules may not apply |
ClaudeBot |
Anthropic | Collecting web content that could contribute to training generative AI models |
Claude-SearchBot |
Anthropic | Improving the quality of search results |
Claude-User |
Anthropic | Access when a Claude user asks a question |
Google-Extended |
A token controlling whether crawled content may be used for training future Gemini models. It doesn't affect inclusion in Google Search, and has no separate HTTP user agent string | |
PerplexityBot |
Perplexity | Surfacing and linking sites in Perplexity search results. Stated as not used for foundation model training |
Perplexity-User |
Perplexity | User-initiated actions |
(Sources at the end. Names and descriptions can change, so check each vendor's latest docs before configuring.)
With that in mind, a policy like "don't use my content for training, but I'm happy to appear in search" can be written like this:
# Opt out of use for generative AI training
User-agent: GPTBot
Disallow: /
User-agent: ClaudeBot
Disallow: /
User-agent: Google-Extended
Disallow: /
# Allow crawlers that power search results
User-agent: OAI-SearchBot
Allow: /
User-agent: Claude-SearchBot
Allow: /
User-agent: PerplexityBot
Allow: /
# Everyone else: business as usual
User-agent: *
Allow: /
Sitemap: https://example.com/sitemap.xml
If you want to allow everything, writing nothing (or Allow: / under User-agent: *) is enough. Which one is right depends on your site's policy.
Things to keep in mind:
- robots.txt is a convention, not enforcement. The protocol spec, RFC 9309, itself says the rules are "not a form of access authorization." Whether to obey is up to the crawler. If something really must stay private, require authentication or don't publish it.
- User-initiated agents (such as
ChatGPT-User) may be treated differently from crawlers regarding robots.txt, as the table above notes. - OpenAI and Perplexity publish their crawlers' IP ranges, so you can also use a WAF to tell real crawlers from requests that merely claim the user agent (see each vendor's docs).
llms.txt: a proposed "front door" for AI
llms.txt is a proposal by Jeremy Howard: a Markdown file summarizing a site for AI agents, placed at /llms.txt or another path.
Important: llms.txt is not a standard. It's a proposal. The spec page itself is worded "We propose...". Also, not every AI service reads it, and I haven't found a primary source confirming that any particular AI always does. I'd be wary of claims like "add it and your ranking or citations will go up" unless evidence is shown.
I still think it's worth trying because it's cheap. Per the spec (llmstxt.org), the file contains, in this order:
- An H1 with the name of the project or site (the only required section)
- A blockquote with a short summary
- Optional: further Markdown (anything except headings)
- Optional: H2 sections containing file lists (
[name](url): notes). By convention, an## Optionalsection holds secondary links that can be skipped when a shorter context is needed
Example (a fictional blog):
# My Blog
> A tech blog about AI agents and app development, based on personal projects.
Articles are in Japanese. Code samples are included in each article.
## Articles
- [What is AIO?](https://example.com/posts/aio.md): An intro to semantic HTML, JSON-LD, robots.txt, and llms.txt
- [Building apps with Expo](https://example.com/posts/expo.md): Notes on solo development with Expo / React Native
## Profile
- [About me](https://example.com/about.md): Author profile
## Optional
- [Archive](https://example.com/archive.md)
The same spec page also proposes serving clean Markdown versions of pages (page.html.md or page.md), without ads or navigation, for the links to point to.
On "access control": llms.txt is not control but guidance. It helps to separate roles: robots.txt for allow/deny signals, llms.txt for guidance.
4. Easy-to-fetch data, APIs, and performance
This last point is unglamorous, but I think it's the most "engineer" part. These are my opinions.
-
Stable URLs and URL design: if the same content is scattered across multiple URLs, readers get confused. Specify a canonical URL (
<link rel="canonical" href="...">). -
Provide machine-oriented exits: a sitemap (
sitemap.xml), RSS/Atom, and, for data that's fine to publish, a JSON API. If there's a route that doesn't require scraping HTML, it's easier for both readers and builders. -
Return correct status codes: avoid returning
200for pages that don't exist (soft 404s). - Be light and fast: heavy pages can affect how much gets crawled and cause timeouts, and they're a worse experience for humans too. Decide what "fast" means by measuring (I won't cite numbers here).
-
Accessibility: keyboard operation,
alttext, labeled form fields. Like semantic HTML, it helps both people and machines.
A checklist to apply to your own site
- [ ] Main content is structured with
main/article/h1… - [ ] Important content is in the HTML before JavaScript runs
- [ ] JSON-LD (Article / SoftwareApplication, etc.) that matches the page content
- [ ] A robots.txt policy that separates training, search, and user-initiated access
- An llms.txt, with the understanding that its effect isn't guaranteed
- [ ] Canonical URL, sitemap, and correct status codes
- [ ] Measured performance and fixed the slow spots
I plan to review my own app landing pages and tech blog against this list.
Conclusion
My concern is this: a site that AI can't find, or can't understand the structure of, is becoming as good as nonexistent. But how each agent reads the web differs and will keep changing. That's why I think the durable move, for humans and AI alike, is not a quick trick for one specific AI, but writing meaning down properly, following standards.
Are you already starting to optimize for AI agents in your projects? I'd love to hear in the comments.
References
- llms.txt spec (llmstxt.org)
- Schema.org / Article / SoftwareApplication
- Google: Intro to structured data
- Google: Article structured data
- Google: Software app structured data
- Google: Crawlers overview / Google's common crawlers (Google-Extended)
- OpenAI: Overview of OpenAI Crawlers
- Anthropic: Does Anthropic crawl data from the web...
- Perplexity: Perplexity Crawlers
- RFC 9309: Robots Exclusion Protocol
-
MDN: Semantics / MDN:
<main>
Top comments (0)