DEV Community

Cover image for How AI reads your website, and what that means for the people who build it

How AI reads your website, and what that means for the people who build it

By Takeshi Yokoyama — Onecarat Labs

Hi. I'm Yokoyama, and I build a local-first AI text editor as a side project, along with a few other experimental tools. Working on them, I keep running into the same question about where the web is going. This post is one observation, plus a small experiment I built to test it — including a Chrome extension you can actually try.

The short version: I think websites will increasingly be read through AI agents, reshaped per reader, on the fly. And once that happens, there's a clear gap between sites that are easy for an AI to read and sites that aren't.

What's starting to happen

Until now, people read websites as websites. You open the top page, follow the menu, read the body, click a button — tracing the path the maker designed.

As local AI and AI agents become normal, that breaks. People stop opening the page directly. They tell an AI what they want — "Can I try this quickly?", "I just want to check it's safe", "Just the gist" — and the AI reads the web and reshapes it into the form that reader wants. What the reader receives is no longer the layout the maker built.

This isn't speculation. The idea that AI generates the interface for the reader already has a name — Generative UI — and it's one of the hottest areas in frontend right now, with Google, Vercel and others building toward it.

But notice who's holding the pen in almost every version of that story: the site, or an AI embedded in an app — something under the maker's control. What I'm looking at is one step past that: a local AI, in the reader's own hands, reshaping any site into that person's preferred form — with no involvement from the maker at all. The initiative moves from the maker to the reader.

The part that nags at me as a builder

I build software too. So this shift nags at me.

A site carries its maker's intent and rights. The order things appear in, what gets emphasized, the tone. Design, copy, flow — all of it is deliberate. Having an AI quietly reorder, rewrite, and re-tone that doesn't sit right with me.

But there's no escaping the other side of it: a site that AI can't read well might as well not exist. Once people search and decide through AI, a site the AI couldn't parse drops out of the running before the reader ever sees it. However carefully you wrote the HTML, if it's too noisy for the AI to read, that care turns into lost opportunity.

So there's a bind. Being reshaped without consent isn't what I want. But not being read means losing the chance entirely. Both are true at once.

Where this conversation already is

I should be honest: I'm not the first to circle this. "Sites should carry machine-readable information for AI agents" is an active conversation abroad, under the name agentic web. Two reference points:

llms.txt (proposed by Jeremy Howard in 2024) — a Markdown file at your site root pointing AI at your important pages. As of 2026, adoption sits around 10%, and the major crawlers barely fetch it — yet it's increasingly framed as a "Business-to-Agent" play, and IDE agents like Cursor and Claude Code do read it routinely. Interestingly, several analyses expect the ecosystem to move from one site-level file toward per-page, machine-readable signals.

NLWeb (Microsoft, announced at Build 2025) — turns a site into something an AI can query in natural language, reusing the Schema.org structured data a site already has. Every NLWeb instance also acts as an MCP server.

So the broad direction — a human web and an AI-facing surface, side by side — isn't a hypothesis anymore. It's already moving.

So what do I think is actually new?

Given all that, here are two gaps I don't think the existing efforts fill — and they're the point of this post.

First, the existing work is mostly about the "make me discoverable" side. llms.txt and NLWeb are largely about getting a cloud-side AI search or agent to find and cite your site — the owner pushing information out. What I'm looking at is the opposite end: a local AI in the reader's hands rebuilding the page's display for that one person's intent. The reader/viewer side. Most of the conversation leans toward "help agents discover the site"; the "the reader's local AI re-renders it per intent" side is still thin.

Second, there's almost no notion of how AI-driven reshaping coexists with the original's authority. In an era where AI summarizes, reorders, and re-tones a site, how do you preserve the maker's intent? That question is barely present in llms.txt or Schema.org. Everyone talks about how the view gets built; almost no one pairs it with "and the original stays sovereign."

My answer to the bind is to split the site into two layers — and hold them both at once.

My answer: two layers, held together

The original (your HTML) is untouchable. The maker controls it completely. Rights and intent live here. If you want it as the maker intended, you read the original.

The derived view is the AI's to build freely. Reorder, summarize, change tone — fine. Because it doesn't touch the original. The original still stands, as the maker made it. The derived view is just one reader's way of looking, for themselves. Don't like it? Read the original.

Here's the thing that makes the free part okay: because the original is preserved, the derived view is free to be whatever it needs to be.

So what does the maker actually do?

This is the practical bit. If the derived view is the AI's to build, the maker doesn't need to design that view. The maker does exactly one thing: make the site easier for AI to read.

Force an AI to parse human-facing HTML and you get misreadings, and instability. Better to hand over the site's information, up front, in a form the AI can read cleanly. So I describe each page as material — not layout — in a JSON file that sits beside the HTML. One extra line in the <head> points to it:

<link rel="ai-source" type="application/json" href="index.ai.json" />
Enter fullscreen mode Exit fullscreen mode
{
  "version": "0.1",
  "site":    { "name": "Onecarat Labs", "url": "https://onecarat.dev", "purpose": "..." },
  "page":    { "path": "index.html", "title": "OnecaratEditor", "type": "product", "summary": "..." },
  "sections": [
    { "id": "download", "title": "Download", "intent": ["try-now"], "content": "Plain-text summary..." }
  ],
  "actions":  [
    { "id": "download-latest", "label": "Download", "url": "...", "intent": ["try-now"] }
  ]
}
Enter fullscreen mode Exit fullscreen mode

The point: this JSON is not an instruction telling the AI how to display anything. It's material, handed over so the AI can read cleanly. The freedom to summarize, reorder, re-tone stays with the reader's AI. The maker doesn't constrain it.

And the result: doing this makes the site easier for AI to read. Easier to read means less lost opportunity. The original HTML stays untouched; whoever wants it reads it. The maker only builds the on-ramp. How the view gets assembled is left to the AI.

It's closer to SEO than to a rendering engine. A site owner never tries to build Google's results page for it. They just want to be found, so they make the site easy to crawl. How the results page is assembled is Google's job. This is that — with an AI agent where the search engine used to be. (And note: the ecosystem is reportedly drifting from one site-level file toward per-page signals anyway — which is exactly the one-JSON-per-page shape here.)

Onecarat Lens — feeling "it got read"

Words only go so far. So I built something you can run: Onecarat Lens, a Chrome extension. It's open source.

🔗 https://github.com/onecarat-labs/onecarat-lens

Lens reads the ai-source a site provides, and lets a local AI — Chrome's built-in Gemini Nano — assemble a view for the reader's intent, painted as an overlay on top of the original. Concretely it:

  1. finds the ai-source link on the page (does nothing if there isn't one),
  2. asks the reader for intent — preset buttons (try-now / safety / overview / details / choose) plus a free-text box,
  3. lets the local AI decide what to keep, how to order it, how to reword it,
  4. paints the result as an overlay that never replaces the original,
  5. and gives a toggle to flip back to the untouched original anytime.

On the same page, "I just want to try it" pushes the download and quick-start to the top; "I want to check it's safe" pushes the local-first, no-cloud, code-signing notes up instead. Same site, different shape — and the source is material the site offered, not a guess scraped from HTML.

What I want you to feel here isn't the polish of the view. It's the fact that the site got read — cleanly, on the reader's terms — and that the original is still right there, one toggle away.

I'll be honest about the rough edges. Gemini Nano is small, so when it's pushed it produces clumsy summaries, or falls back. Lens shows that fallback openly rather than hiding it. That's deliberate. The point of this experiment is honesty about what local AI can and can't do yet.

A note on where this came from

This whole idea came out of a wrong turn. An earlier version of Lens read a site's raw HTML and let the AI rebuild it directly. And it felt like overriding the maker's intent. The fix wasn't to make the AI more polite about it. It was to let the original stay sovereign, and have the site offer material for the AI on its own terms. That realization is what ai-source is.

Wrapping up

"HTML for humans, JSON for AI" is no longer just my hypothesis — it's already moving, as llms.txt and NLWeb. What I think is new is two things: not the discovery side but the reader's local AI re-rendering per intent, and a maker who declares nothing about the view but simply makes the site readable, paired with an original that stays sovereign.

Local AI will only get more practical. But it holds no fresh information, so it'll be asked to go and read the web. And today's human-facing web is too noisy for it. So the site offers something readable; the original stays whole; the reader sees it through their own local AI, in the shape that fits them.

Onecarat Lens is a small experiment to test that future. It doesn't replace the web. It adds an on-ramp for AI to the human-facing HTML — opened from the maker's side, deliberately, on the maker's terms.

If local AI agents become normal, sites will be read by AI too. Whether you've made yours readable, by then, is the thing that will matter.


Code: github.com/onecarat-labs/onecarat-lens · Built by Onecarat Labs

Top comments (19)

Collapse
 
alexshev profile image
Alex Shev

This matches what I'm seeing too: the website is becoming both a human interface and a machine-readable corpus. The practical implication is less about hiding content from AI and more about making the important state, services, pricing, locations, and trust signals unambiguous.

If the page only works visually, agents will reconstruct it badly.

Collapse
 
onecaratlabs profile image
Onecarat Labs/Takeshi Yokoyama

Thanks Alex! "Human interface and machine-readable corpus at the same time" — that's a sharper way to say what I was getting at. And agreed: the point isn't hiding from AI, it's removing ambiguity. A page that only works visually gets reconstructed badly, which is exactly the gap I wanted to close.

Collapse
 
alexshev profile image
Alex Shev

Exactly. The practical test I keep coming back to is: if you remove the styling, can a crawler or agent still recover the intent of the page correctly?

Headings, internal links, examples, author/entity signals, product/service boundaries — those are not just SEO decoration anymore. They are the contract between the human page and the machine reader.

Thread Thread
 
onecaratlabs profile image
Onecarat Labs/Takeshi Yokoyama

Totally agree, Alex — if the structure is right, a capable agent can recover the intent. Your point is solid, and honestly that's the world I'd expect with strong models.

The one extra angle I keep thinking about: when the agent is a local one — small model, limited resources, running on the user's own device — "recover the intent from structure" gets expensive. Re-parsing a full page to reconstruct meaning costs context and compute a small local model doesn't really have. That's where ai-source earns its place: it lets a weak local agent grasp the page cheaply, without reverse-engineering it.

But I'll be honest about the flip side — as local models get stronger, that cost argument weakens. At some point they may just recover the intent fine on their own, and your approach becomes all you need. So I partly see ai-source as a bridge for the low-resource era, not necessarily a permanent layer. Curious whether you'd bet the same way.

Thread Thread
 
alexshev profile image
Alex Shev

I would bet on a middle ground.

Strong models will get better at reconstructing intent, but I do not think every agent should spend tokens doing archaeology on every page. For local or on-device agents especially, the cheaper path is still: give them a small explicit semantic layer so they can decide fast whether the page is relevant before doing deeper reading.

So I see ai-source less as a permanent replacement for good structure and more as an intent cache/index. Good HTML and content should stay the source of truth. The machine-readable layer should make the obvious things cheap: what the page is about, who it is for, key entities, actions, constraints, and freshness.

If models get much stronger, that layer becomes less critical for comprehension. But it can still matter for cost, latency, and consistency.

Thread Thread
 
onecaratlabs profile image
Onecarat Labs/Takeshi Yokoyama

"Intent cache / index" and "make the obvious things cheap" — that's a better framing of ai-source than I had myself. I'm stealing both.

And I think we've landed in the same place: HTML and content stay the source of truth, the machine layer just makes the obvious stuff cheap to grasp — and even when models get strong enough that comprehension no longer needs it, cost, latency and consistency keep it useful. That's almost exactly the "it survives in some form" intuition I had but couldn't put into words. Thanks for sharpening it — genuinely the most useful exchange I've had on this idea so far.

Thread Thread
 
alexshev profile image
Alex Shev

Steal both. That is the best outcome of this kind of thread.

I think the durable version is exactly what you said: HTML/content remains the source of truth, and the machine layer is a cheap index over intent. Even if stronger models can reconstruct it, the index still wins on cost, latency, and consistency.

Thread Thread
 
onecaratlabs profile image
Onecarat Labs/Takeshi Yokoyama

Agreed on all counts — I think we've landed it: content as source of truth, index as the cheap front door.

Thanks for the thread, Alex. There's probably a follow-up post in here somewhere.

Thread Thread
 
alexshev profile image
Alex Shev

Yes, exactly. The follow-up post is probably the practical version of the thread: how to make the source of truth readable to humans first, then expose just enough structure for AI/search systems to trust it.

The cheap front door only works if the house behind it is organized.

Thread Thread
 
onecaratlabs profile image
Onecarat Labs/Takeshi Yokoyama

Thank you, Alex. That kind of encouragement genuinely keeps an experiment moving.

Here's what I want to try next: a tiny discovery layer for local agents. Less "a Google for AI," more an index that only ever returns pointers. For each site it hands back two things — a light summary (just enough to decide whether to read it) and the URL of that site's AI layer (its ai-source). No content. The summary is a discovery key, not the thing you display.

The Lens takes those pointers, goes and reads the actual origins it picked, and fuses several of them into a single view shaped around the reader's intent — a page that exists only for you, assembled from sources that each stay sovereign where they live.

The shift I'm chasing is the unit of reconstruction. Today's AI browsers rewrite one page to make it readable. I want that unit to move from the page to the reader: many origins, one personal view. The rule I'm holding is that the central layer never holds what gets shown — it only helps you find what to read. The moment it caches the content itself, it quietly becomes a second source of truth, and the whole point collapses.

That's the experiment. I'll write it up somewhere down the line.

Thread Thread
 
alexshev profile image
Alex Shev

That pointer-only discovery layer is the interesting part. If the index returns just enough to decide where to read next, the original sites keep authority and the agent still gets a useful map. The hard product line is making summaries helpful without becoming a shadow copy.

Thread Thread
 
alexshev profile image
Alex Shev

That pointer-only discovery layer is the right shape. It keeps the answer engine from pretending it owns the source, while still making the useful next read discoverable.

Collapse
 
itskondrat profile image
Mykola Kondratiuk

been running into this with internal tooling - when you prompt an agent to read a project wiki, structure matters way more than content. docs not designed for skim-reading do even worse for agents than for humans.

Collapse
 
onecaratlabs profile image
Onecarat Labs/Takeshi Yokoyama

Internal tooling is where this bites hardest, I think — especially if you're running smaller or local models for it. A frontier model can sometimes brute-force its way through a messy wiki page; a small on-device agent just can't afford that archaeology. Structure isn't an optimization there, it's a requirement.

Thanks for adding the internal-docs angle — the thread had been mostly about public websites.

Collapse
 
itskondrat profile image
Mykola Kondratiuk

the model tier shapes the structure requirement in both directions — frontier models can approximate intent from loose docs, which makes bad habits survivable. on-device models surface the debt immediately. the structure problem was always there; small models just have no working memory to hide it in.

Thread Thread
 
onecaratlabs profile image
Onecarat Labs/Takeshi Yokoyama

Right — and "no working memory to hide it in" is the whole thing, I think. A frontier model spends its slack reconstructing intent and you never see the bill. A small model has no slack to spend, so the bill shows up immediately.

That's actually why the cheap-index idea keeps pulling at me. It isn't about hiding the debt — that's the frontier move, and that's exactly the bad habit. It's about not making a small model burn its scarce budget re-deriving what was obvious to begin with. Spend the working memory on the content, not on guessing the structure.

So the debt stays visible. The index just means the small model doesn't have to carry it.

Thread Thread
 
itskondrat profile image
Mykola Kondratiuk

The enforced bill is the useful part. Frontier models absorb ambiguity and you pay later when something guessed wrong compounds. The cheap-index forces the structure to be explicit — which is where the real design work is anyway.

Collapse
 
devitoben profile image
Ava Bennet

Spot on, if AI agents become a standard way of interacting with the web, allowing AI access to your website with ease and having a clean UX and article structure will be really important for visibility in the new AI search era.

Collapse
 
onecaratlabs profile image
Onecarat Labs/Takeshi Yokoyama

Exactly — visibility is shifting from "can you rank?" to "can you be read and cited?" If an agent can't cleanly extract what you do, who it's for, and why you're credible, you're effectively invisible no matter how good the content is.

The nice part: most of what helps agents (clean structure, unambiguous intent) helps human readers too.