DEV Community

Cover image for How AI reads your website, and what that means for the people who build it

How AI reads your website, and what that means for the people who build it

Onecarat Labs/Takeshi Yokoyama on May 31, 2026

By Takeshi Yokoyama — Onecarat Labs Hi. I'm Yokoyama, and I build a local-first AI text editor as a side project, along with a few other experimen...
Collapse
 
alexshev profile image
Alex Shev

This matches what I'm seeing too: the website is becoming both a human interface and a machine-readable corpus. The practical implication is less about hiding content from AI and more about making the important state, services, pricing, locations, and trust signals unambiguous.

If the page only works visually, agents will reconstruct it badly.

Collapse
 
onecaratlabs profile image
Onecarat Labs/Takeshi Yokoyama

Thanks Alex! "Human interface and machine-readable corpus at the same time" — that's a sharper way to say what I was getting at. And agreed: the point isn't hiding from AI, it's removing ambiguity. A page that only works visually gets reconstructed badly, which is exactly the gap I wanted to close.

Collapse
 
alexshev profile image
Alex Shev

Exactly. The practical test I keep coming back to is: if you remove the styling, can a crawler or agent still recover the intent of the page correctly?

Headings, internal links, examples, author/entity signals, product/service boundaries — those are not just SEO decoration anymore. They are the contract between the human page and the machine reader.

Thread Thread
 
onecaratlabs profile image
Onecarat Labs/Takeshi Yokoyama

Totally agree, Alex — if the structure is right, a capable agent can recover the intent. Your point is solid, and honestly that's the world I'd expect with strong models.

The one extra angle I keep thinking about: when the agent is a local one — small model, limited resources, running on the user's own device — "recover the intent from structure" gets expensive. Re-parsing a full page to reconstruct meaning costs context and compute a small local model doesn't really have. That's where ai-source earns its place: it lets a weak local agent grasp the page cheaply, without reverse-engineering it.

But I'll be honest about the flip side — as local models get stronger, that cost argument weakens. At some point they may just recover the intent fine on their own, and your approach becomes all you need. So I partly see ai-source as a bridge for the low-resource era, not necessarily a permanent layer. Curious whether you'd bet the same way.

Thread Thread
 
alexshev profile image
Alex Shev

I would bet on a middle ground.

Strong models will get better at reconstructing intent, but I do not think every agent should spend tokens doing archaeology on every page. For local or on-device agents especially, the cheaper path is still: give them a small explicit semantic layer so they can decide fast whether the page is relevant before doing deeper reading.

So I see ai-source less as a permanent replacement for good structure and more as an intent cache/index. Good HTML and content should stay the source of truth. The machine-readable layer should make the obvious things cheap: what the page is about, who it is for, key entities, actions, constraints, and freshness.

If models get much stronger, that layer becomes less critical for comprehension. But it can still matter for cost, latency, and consistency.

Thread Thread
 
onecaratlabs profile image
Onecarat Labs/Takeshi Yokoyama

"Intent cache / index" and "make the obvious things cheap" — that's a better framing of ai-source than I had myself. I'm stealing both.

And I think we've landed in the same place: HTML and content stay the source of truth, the machine layer just makes the obvious stuff cheap to grasp — and even when models get strong enough that comprehension no longer needs it, cost, latency and consistency keep it useful. That's almost exactly the "it survives in some form" intuition I had but couldn't put into words. Thanks for sharpening it — genuinely the most useful exchange I've had on this idea so far.

Collapse
 
itskondrat profile image
Mykola Kondratiuk

been running into this with internal tooling - when you prompt an agent to read a project wiki, structure matters way more than content. docs not designed for skim-reading do even worse for agents than for humans.

Collapse
 
onecaratlabs profile image
Onecarat Labs/Takeshi Yokoyama

Internal tooling is where this bites hardest, I think — especially if you're running smaller or local models for it. A frontier model can sometimes brute-force its way through a messy wiki page; a small on-device agent just can't afford that archaeology. Structure isn't an optimization there, it's a requirement.

Thanks for adding the internal-docs angle — the thread had been mostly about public websites.

Collapse
 
itskondrat profile image
Mykola Kondratiuk

the model tier shapes the structure requirement in both directions — frontier models can approximate intent from loose docs, which makes bad habits survivable. on-device models surface the debt immediately. the structure problem was always there; small models just have no working memory to hide it in.

Thread Thread
 
onecaratlabs profile image
Onecarat Labs/Takeshi Yokoyama

Right — and "no working memory to hide it in" is the whole thing, I think. A frontier model spends its slack reconstructing intent and you never see the bill. A small model has no slack to spend, so the bill shows up immediately.

That's actually why the cheap-index idea keeps pulling at me. It isn't about hiding the debt — that's the frontier move, and that's exactly the bad habit. It's about not making a small model burn its scarce budget re-deriving what was obvious to begin with. Spend the working memory on the content, not on guessing the structure.

So the debt stays visible. The index just means the small model doesn't have to carry it.

Thread Thread
 
itskondrat profile image
Mykola Kondratiuk

The enforced bill is the useful part. Frontier models absorb ambiguity and you pay later when something guessed wrong compounds. The cheap-index forces the structure to be explicit — which is where the real design work is anyway.

Collapse
 
devitoben profile image
Ava Bennet

Spot on, if AI agents become a standard way of interacting with the web, allowing AI access to your website with ease and having a clean UX and article structure will be really important for visibility in the new AI search era.

Collapse
 
onecaratlabs profile image
Onecarat Labs/Takeshi Yokoyama

Exactly — visibility is shifting from "can you rank?" to "can you be read and cited?" If an agent can't cleanly extract what you do, who it's for, and why you're credible, you're effectively invisible no matter how good the content is.

The nice part: most of what helps agents (clean structure, unambiguous intent) helps human readers too.