DEV Community

kohei matsunobu
kohei matsunobu

Posted on Originally published at zenn.dev

Building Sites That Both Humans and AI Agents Can Read: A Practical Intro to AIO

This article was originally published in Japanese on Zenn.

Introduction

Hi, I'm Kohei Matsunobe, a second-year student at ZEN University studying Intelligent Informatics (online). I build AI agents and apps as personal projects, including an iOS app called Kamimegu and apps built with Expo / React Native.

I recently wrote this on LinkedIn:

The skills web engineers need from now on are not only "building sites that are easy for humans to read." Building sites that AI agents (such as GrokBot and Muse) can correctly recognize is becoming essential.

When you build AI agents, you notice that an agent is also a reader of the web. We are moving from a world where people click links in search results to one where agents collect and interpret information on the web and present answers or actions. In that world, the question for site builders is not only how a page looks, but whether a machine can read its structure without getting it wrong. I think of this as DX for AI: Data Experience.

I call this viewpoint AIO (AI Optimization). In this article I turn the four points from my post into minimal code:

  1. Semantic HTML
  2. Structured data (JSON-LD)
  3. Access control with AI crawlers in mind (robots.txt), and llms.txt
  4. Easy-to-fetch data, APIs, and performance

A caveat up front: this is not a "do this and AI will definitely pick you up" guide. I haven't verified from primary sources how any specific AI agent parses a site, so I won't claim anything about that. What follows is based on official documentation and specs: how to make pages machine-readable, and how to signal your intent to crawlers.

1. Semantic HTML: let the tags carry the meaning

The problem

Two pages can look identical, yet a page made only of divs doesn't tell a machine which part is the heading, the main content, or the navigation. A human can tell by looking. Something reading the HTML has far fewer clues beyond the tags.

Before

<div class="header">
  <div class="logo">My Blog</div>
  <div class="menu">
    <div><a href="/">Home</a></div>
    <div><a href="/posts">Posts</a></div>
  </div>
</div>
<div class="content">
  <div class="title">What is AIO?</div>
  <div class="date">2026-11-18</div>
  <div class="text">Body text...</div>
</div>
<div class="footer">© 2026</div>
Enter fullscreen mode Exit fullscreen mode

After

<header>
  <p class="logo">My Blog</p>
  <nav aria-label="Main">
    <ul>
      <li><a href="/">Home</a></li>
      <li><a href="/posts">Posts</a></li>
    </ul>
  </nav>
</header>
<main>
  <article>
    <h1>What is AIO?</h1>
    <p><time datetime="2026-11-18">November 18, 2026</time></p>
    <p>Body text...</p>
  </article>
</main>
<footer>© 2026</footer>
Enter fullscreen mode Exit fullscreen mode

Key points:

  • Put the page's primary content in <main> (one per page).
  • Use <article> for self-contained pieces and <nav> for navigation.
  • Use headings in order (h1 → h2 → h3) without skipping levels. Don't pick a heading tag for its size; use CSS for that.
  • Use <time datetime="..."> for dates and meaningful alt text for images.

This is also standard accessibility practice (screen readers). Here, "easy for humans to use" and "easy for machines to read" turn out to be the same work.

References: MDN: Semantics, MDN: <main>

Side note: if your content only appears in the HTML after JavaScript runs, clients that don't execute JavaScript will see nothing. Whether a client runs JavaScript is up to that client, so it's safer to include important content in the initial HTML response (SSR / SSG). If you export Expo or React Native code to the web, keep this in mind for the pages you want to be read, such as a product page.

2. Structured data (JSON-LD): add meaning in a machine-readable form

Semantic HTML describes a document's skeleton. Structured data states what a page is about (an article, an app, who wrote it) using the Schema.org vocabulary. Google recommends JSON-LD as the format (Google: Intro to structured data).

An article page (Article)

<script type="application/ld+json">
{
  "@context": "https://schema.org",
  "@type": "Article",
  "headline": "What is AIO?",
  "datePublished": "2026-11-18T09:00:00+09:00",
  "dateModified": "2026-11-18T09:00:00+09:00",
  "author": [
    {
      "@type": "Person",
      "name": "Kohei Matsunobe",
      "url": "https://zenn.dev/koheip"
    }
  ],
  "image": ["https://example.com/images/aio-cover.png"]
}
</script>
Enter fullscreen mode Exit fullscreen mode

Google's Article documentation says there are no required properties; you add the recommended ones that apply (Google: Article structured data).

An app landing page (SoftwareApplication)

For an app's landing page, SoftwareApplication fits. These are dummy values:

<script type="application/ld+json">
{
  "@context": "https://schema.org",
  "@type": "SoftwareApplication",
  "name": "Sample App",
  "operatingSystem": "iOS",
  "applicationCategory": "LifestyleApplication",
  "offers": {
    "@type": "Offer",
    "price": "0",
    "priceCurrency": "JPY"
  }
}
</script>
Enter fullscreen mode Exit fullscreen mode

Two notes:

  • Keep the values consistent with what the page actually says. Don't put ratings or prices only in structured data when they don't appear on the page.
  • If you want Google's software app rich result, there are required properties of its own: name, offers.price, and aggregateRating or review (Google: Software app). If no real ratings or reviews exist, don't invent them. As a Schema.org vocabulary, the minimal form above is still valid.

You can browse the available types and properties at Schema.org/Article and Schema.org/SoftwareApplication. Adding structured data doesn't guarantee that any particular AI agent will use it. For Google Search features, the docs above are the primary source.

3. Crawler-aware access control: robots.txt and llms.txt

robots.txt: signal your intent per crawler

What matters here is that crawlers have different purposes. The vendors' own docs draw these distinctions:

User agent / token Vendor Purpose as stated in the docs
GPTBot OpenAI Crawling content that may be used to train generative AI foundation models
OAI-SearchBot OpenAI Surfacing sites in ChatGPT's search features
ChatGPT-User OpenAI User-initiated actions. Not automatic crawling; the docs say robots.txt rules may not apply
ClaudeBot Anthropic Collecting web content that could contribute to training generative AI models
Claude-SearchBot Anthropic Improving the quality of search results
Claude-User Anthropic Access when a Claude user asks a question
Google-Extended Google A token controlling whether crawled content may be used for training future Gemini models. It doesn't affect inclusion in Google Search, and has no separate HTTP user agent string
PerplexityBot Perplexity Surfacing and linking sites in Perplexity search results. Stated as not used for foundation model training
Perplexity-User Perplexity User-initiated actions

(Sources at the end. Names and descriptions can change, so check each vendor's latest docs before configuring.)

With that in mind, a policy like "don't use my content for training, but I'm happy to appear in search" can be written like this:

# Opt out of use for generative AI training
User-agent: GPTBot
Disallow: /

User-agent: ClaudeBot
Disallow: /

User-agent: Google-Extended
Disallow: /

# Allow crawlers that power search results
User-agent: OAI-SearchBot
Allow: /

User-agent: Claude-SearchBot
Allow: /

User-agent: PerplexityBot
Allow: /

# Everyone else: business as usual
User-agent: *
Allow: /

Sitemap: https://example.com/sitemap.xml
Enter fullscreen mode Exit fullscreen mode

If you want to allow everything, writing nothing (or Allow: / under User-agent: *) is enough. Which one is right depends on your site's policy.

Things to keep in mind:

  • robots.txt is a convention, not enforcement. The protocol spec, RFC 9309, itself says the rules are "not a form of access authorization." Whether to obey is up to the crawler. If something really must stay private, require authentication or don't publish it.
  • User-initiated agents (such as ChatGPT-User) may be treated differently from crawlers regarding robots.txt, as the table above notes.
  • OpenAI and Perplexity publish their crawlers' IP ranges, so you can also use a WAF to tell real crawlers from requests that merely claim the user agent (see each vendor's docs).

llms.txt: a proposed "front door" for AI

llms.txt is a proposal by Jeremy Howard: a Markdown file summarizing a site for AI agents, placed at /llms.txt or another path.

Important: llms.txt is not a standard. It's a proposal. The spec page itself is worded "We propose...". Also, not every AI service reads it, and I haven't found a primary source confirming that any particular AI always does. I'd be wary of claims like "add it and your ranking or citations will go up" unless evidence is shown.

I still think it's worth trying because it's cheap. Per the spec (llmstxt.org), the file contains, in this order:

  • An H1 with the name of the project or site (the only required section)
  • A blockquote with a short summary
  • Optional: further Markdown (anything except headings)
  • Optional: H2 sections containing file lists ([name](url): notes). By convention, an ## Optional section holds secondary links that can be skipped when a shorter context is needed

Example (a fictional blog):

# My Blog

> A tech blog about AI agents and app development, based on personal projects.

Articles are in Japanese. Code samples are included in each article.

## Articles

- [What is AIO?](https://example.com/posts/aio.md): An intro to semantic HTML, JSON-LD, robots.txt, and llms.txt
- [Building apps with Expo](https://example.com/posts/expo.md): Notes on solo development with Expo / React Native

## Profile

- [About me](https://example.com/about.md): Author profile

## Optional

- [Archive](https://example.com/archive.md)
Enter fullscreen mode Exit fullscreen mode

The same spec page also proposes serving clean Markdown versions of pages (page.html.md or page.md), without ads or navigation, for the links to point to.

On "access control": llms.txt is not control but guidance. It helps to separate roles: robots.txt for allow/deny signals, llms.txt for guidance.

4. Easy-to-fetch data, APIs, and performance

This last point is unglamorous, but I think it's the most "engineer" part. These are my opinions.

  • Stable URLs and URL design: if the same content is scattered across multiple URLs, readers get confused. Specify a canonical URL (<link rel="canonical" href="...">).
  • Provide machine-oriented exits: a sitemap (sitemap.xml), RSS/Atom, and, for data that's fine to publish, a JSON API. If there's a route that doesn't require scraping HTML, it's easier for both readers and builders.
  • Return correct status codes: avoid returning 200 for pages that don't exist (soft 404s).
  • Be light and fast: heavy pages can affect how much gets crawled and cause timeouts, and they're a worse experience for humans too. Decide what "fast" means by measuring (I won't cite numbers here).
  • Accessibility: keyboard operation, alt text, labeled form fields. Like semantic HTML, it helps both people and machines.

A checklist to apply to your own site

  • [ ] Main content is structured with main / article / h1…
  • [ ] Important content is in the HTML before JavaScript runs
  • [ ] JSON-LD (Article / SoftwareApplication, etc.) that matches the page content
  • [ ] A robots.txt policy that separates training, search, and user-initiated access
  • An llms.txt, with the understanding that its effect isn't guaranteed
  • [ ] Canonical URL, sitemap, and correct status codes
  • [ ] Measured performance and fixed the slow spots

I plan to review my own app landing pages and tech blog against this list.

Conclusion

My concern is this: a site that AI can't find, or can't understand the structure of, is becoming as good as nonexistent. But how each agent reads the web differs and will keep changing. That's why I think the durable move, for humans and AI alike, is not a quick trick for one specific AI, but writing meaning down properly, following standards.

Are you already starting to optimize for AI agents in your projects? I'd love to hear in the comments.

References

Top comments (0)