DEV Community: Ali Habibzadeh

Is GEO only in our heads, or something real?

Ali Habibzadeh — Wed, 23 Sep 2026 23:43:46 +0000

I was deep in thought about how much of a model hallucination is RLHF-based assumption in the absence of training data or context. Pretraining gives the model the ability to confabulate. Post-training often influences whether it chooses to confabulate rather than say "I don't know."

A base language model is trained to predict plausible continuations. If the evidence needed to answer is absent from its weights or context, there's no fundamental mechanism in next-token prediction that says "stop, evidence unavailable." It still has a probability distribution over tokens, so it generates the most plausible continuation. Hallucinations therefore exist without RLHF.

TruthfulQA demonstrated this with pre-RLHF base models such as GPT-3, GPT-2 and GPT-Neo/J: they confidently reproduced misconceptions learned from text, and the larger models were less truthful, not more.

But this is a slightly narrow point of view for web people like us. For us, when we think of hallucination we think of a model telling an untruth. Which the above is really a subset of.

Seen in the broader sense, hallucination decomposes like this:

Wrong information encoded during pretraining. The model learned a false claim because the corpus contained it, or because statistical compression distorted the underlying facts.
Imperfect retrieval from model weights. The correct information is effectively "in there," but generation retrieves the wrong association: wrong year, wrong person, conflated events.
Inference errors. The premises are available, the model reasons incorrectly.
Confabulation under missing information. The model lacks the fact and constructs the most plausible answer anyway.
Post-training induced behaviour. SFT, RLHF and preference tuning make 1 to 4 more or less likely to be verbalised as an assertion, depending on whether the learned behaviour rewards answering, agreeing, sounding decisive, abstaining or checking uncertainty.

I know what you're thinking, this is still narrow. We don't interact with bare models like this. We interact with fully fledged ecosystems with so many moving parts: model, system prompt, memory, retrieval, tools, planners, execution loop, verifiers, reward shaping, UI constraints, external state. What fancy people call harnesses.

In that environment an untrue response can arrive for a much wider variety of reasons:

The base model invents a fact.
Retrieval returns the wrong document, stale information, or a semantically similar but incorrect result.
The model misreads correct retrieved evidence.
Tool selection is wrong: it searches the web when it should query an MCP server, queries the wrong database, or fails to use a tool at all.
The planner makes an unsupported assumption early, and every later step treats it as established fact.
Memory injects an incorrect prior fact.
A tool succeeds partially, but the agent interprets partial output as complete.
The agent loses provenance across steps and can no longer distinguish observed facts from inferred facts.
A verifier checks linguistic consistency rather than correspondence with reality.
The system prompt or reward function pressures the agent to "finish the task," so it bridges missing state with assumptions.
One agent invents something and another agent treats that output as evidence.
The final response generator turns an uncertain intermediate state into a declarative statement.

Why even think about any of this? Well, because if you're to believe GEO is a real discipline, you have to believe you can impact some of these factors in a controlled way as an outsider. Otherwise there'll be nothing to optimise, right?

The important distinction is control over the model versus control over the evidence environment the model operates in.

What can we do then?

Leverage is qualified the same way throughout: where a remediation requires you to operate a machine endpoint, the leverage is conditional on your doing so. Rows 4, 7 and 8 all carry that condition, and an earlier version of this table applied it only to row 4.

#	Failure	Business leverage	Evidence	What the business can actually do
1	Base model invents a fact	Low	Strong that it happens; weak that businesses can fix weights	Improve the public factual corpus; correct authoritative third parties; allow training crawlers if you want to. Don't promise that publishing something changes model weights.
2	Wrong or stale retrieval	Very high	Strong	Crawlability, indexing, canonical URLs, freshness signals, authoritative pages, remove stale duplicates, correct third-party sources
3	Model misreads correct evidence	High	Strong to moderate	Make facts explicit, atomic, unambiguous, scoped, easy to extract. Reduce contradictory and verbose pages.
4	Wrong tool selected	High, if you expose tools	Strong and growing	MCP and API discovery, precise tool descriptions, narrow responsibilities, obvious capability names, good schemas
5	Planner invents assumption	Medium-high	Strong underlying evidence	Publish direct answers to likely ambiguous questions, and explicit negative facts, so retrieval resolves rather than infers
6	Incorrect memory	Low externally	Mixed	Make the current authoritative state easy to retrieve so live evidence can override memory. You can't normally edit another provider's memory.
7	Partial tool result treated as complete	Very high, if you expose tools	Engineering evidence stronger than GEO research	Design API and MCP responses with pagination, completeness, timestamps, status, errors, coverage boundaries
8	Provenance disappears	Medium-high, if you return machine-readable facts	Strong problem evidence	Attach source URLs, IDs, timestamps and claim provenance to every fact you return
9	Bad verifier	Low	Strong that verification design matters	Provide ground-truth endpoints a verifier could use. You don't control whether third parties use them.
10	Reward or system prompt encourages guessing	Very low	Strong phenomenon, outside business control	Almost nothing externally, except make uncertainty unnecessary by improving evidence availability
11	Agent trusts another agent's fabrication	Medium	Growing evidence	Provide provenance-rich primary evidence so downstream agents can check upstream claims
12	Final generator converts uncertainty into fact	Low-medium	Strong phenomenon	Make evidence decisive and machine-readable; you can't control another system's final-generation policy

2, 3, 4, 5, 7 and 8. Those are the genuinely externally addressable hallucination surfaces. And this is much bigger than conventional GEO.

What follows adds something the argument didn't have when I first wrote it: measurements of how those six surfaces actually look across the web, rather than only arguments that they matter.

Wrong or stale retrieval is probably the biggest opportunity

RAG exists precisely because supplying current external information reduces dependence on faulty parametric knowledge. Models frequently update an incorrect parametric belief when given correct external evidence, although strong priors can still cause resistance. Retrieval quality matters enormously: wrong, irrelevant and conflicting documents measurably degrade answer correctness.

A business can directly work on this. Suppose models incorrectly say "Lumar doesn't support JavaScript rendering." You want a canonical URL that states the fact plainly:

Does Lumar crawl JavaScript websites?
Yes. Lumar renders pages using Chromium and crawls content generated by client-side JavaScript.

rather than expecting the model to derive it from 1,800 words of marketing copy. Then make sure that page is crawlable, internally linked, canonical, indexed, current, server-rendered or otherwise trivially extractable, and not contradicted by three old documentation pages.

And here's the part I can now measure rather than just assert (Agentic readiness of the web, corpus scan). The retrieval substrate isn't in good shape. Of the 88,484 publishing hosts, 61,587 publish an XML sitemap and 21,448 of them break the format, about 35%. The single most common fault anywhere in the study is a sitemap sitting in a subfolder that lists URLs outside it: 14,176 sites, 23% of everyone with a sitemap. Google reads those URLs anyway if you submitted the file through Search Console, which is why so many teams have never seen it break. Hand the same file to anything that isn't Google and the out-of-scope entries are dropped.

The sitemap protocol was agreed in 2005. If the oldest, most boring piece of retrieval plumbing on the web is broken on a third of sites that bother to publish it, the idea that the evidence environment is already optimised and only model behaviour is left to tune doesn't survive contact with the data.

Basic freshness engineering matters too: Google [explicitly uses trustworthy lastmod values (https://developers.google.com/search/blog/2023/06/sitemaps-lastmod-ping) as a recrawling signal.

One correction worth making precisely, in an article about precision. For ChatGPT Search, OpenAI says that "sites that are opted out of OAI-SearchBot will not be shown in ChatGPT search answers, though can still appear as navigational links," and that they "recommend allowing OAI-SearchBot." That's narrower than "must be allowed to be eligible": you lose the answers, not the link. They separately state GPTBot "is used to crawl content that may be used in training our generative AI foundation models," which is the distinction that matters here.

It's also a decision very few sites have taken deliberately (corpus scan). 33,626 sites name at least one AI crawler in robots.txt, and only 4,289 of them name OAI-SearchBot at all, about one in eight. A caveat I won't skip:
naming a crawler isn't blocking it, and a crawl can't see which way a rule points. What the number shows is how few sites have made any explicit decision about the one bot that governs appearing in ChatGPT answers.

None of this is speculative "LLM SEO." It's improving the evidence retrieval systems have available.

Correct evidence that the model misinterprets is also addressable

This is where I'd invest substantially. Models do worse as irrelevant context increases, and evidence placement affects factual verification.

So stop asking "is this information somewhere on our site?" and start measuring:

Can a machine retrieve the smallest sufficient passage that answers the question without inference?

Those are very different standards.

Bad:

Our platform empowers modern enterprise teams through an advanced rendering infrastructure designed to help
organizations understand experiences across increasingly dynamic digital environments...

Good:

Does Lumar crawl JavaScript websites?
Yes. Lumar can render pages using Chromium and crawl content generated by client-side JavaScript.

This matters because you're reducing the amount of reasoning between evidence and claim. I'd call the metric something like inference distance: how many unsupported transformations are required to get from the source material to the desired factual answer. You want that close to zero.

Tool selection is increasingly something businesses can influence

This is where the agentic ecosystem gets interesting, and where I have the sharpest numbers.

MCP and tool-selection research shows measurable differences in models' susceptibility to misleading tool descriptions and overlapping capabilities. A 2026 study ran 8,640 agent runs and found substantial variation in wrong-tool susceptibility depending on tool design and model capability.

So if you expose:

{ "name": "query", "description": "Query the system" }

versus:

{
  "name": "salesforce_get_account_by_id",
  "description": "Authoritative source for Salesforce CRM account records. Returns the current
                  account owner, tier and renewal date for one account ID. Not for opportunities,
                  contacts or historical snapshots."
}

the latter makes epistemically correct tool selection much easier. You can't force an agent to pick your tool. But you can remove ambiguity from the decision.

And this is where the numbers get embarrassing (corpus scan). Across the 88,484 publishing hosts:

	Sites
Publish an OpenAPI description	497
...with a value whose type is never stated	176
...with an operation that has no `operationId`	112
...with two operations sharing one `operationId`	66
Publish an MCP server card	368
...whose card doesn't say where the server is	40
...whose endpoint doesn't speak MCP	19

Look at row three again. 112 of 497 machine-readable API descriptions give an agent nothing stable to name a call by (corpus scan). That's not a theoretical concern about tool ontology. It's the concrete failure this section argues about, already present in 23% of the sites that have gone to the trouble of publishing a contract at all. And 40 server cards advertise an MCP server without saying where it is.

For businesses this suggests a new optimisation surface:

Don't merely expose an MCP server. Engineer its ontology so an agent can reliably determine when it is the > authoritative source.

Unsupported planner assumptions are surprisingly addressable

Consider: "Why is Lumar more expensive than Screaming Frog?"

If neither company has published meaningful evidence, an agent will generate something plausible. Possibly complete fiction. A company can anticipate those inference gaps and publish the answer:

How does Lumar's pricing compare with Screaming Frog's?
They are priced for different jobs. Screaming Frog licenses a desktop crawler per seat. Lumar is billed on > crawl volume and retained history across a whole estate. A single-site audit is cheaper in Screaming Frog; > continuous monitoring of many domains is what Lumar is priced for.

Or, importantly, publish the negative:

Does Lumar offer a free tier?
No. Lumar does not offer a free or perpetually free plan.

Negative knowledge is particularly useful. Companies publish enormous amounts of X does Y and very little X does **not* do Y*. False assumptions live precisely in that missing space.

Provenance may be the biggest long-term opportunity

Generative search research has repeatedly found provenance problems. Liu, Zhang and Liang's Evaluating Verifiability in Generative Search Engines
(Findings of EMNLP 2023) audited four engines by human evaluation and found only 51.5% of generated sentences fully supported by their citations, and only 74.5% of citations actually supporting the sentence they were attached to.

That study is worth citing carefully rather than waving at. It is from 2023, and it audited Bing Chat, NeevaAI, perplexity.ai and YouChat. NeevaAI no longer exists. It establishes that the problem is real and was severe; it doesn't describe 2026 systems, and I haven't found an equivalently rigorous replication since.

For a business-controlled machine endpoint, I'd therefore return facts more like:

{
  "claim": "Lumar renders JavaScript using Chromium",
  "value": true,
  "source": "https://www.lumar.io/docs/crawler/javascript-rendering/",
  "asOf": "2026-09-01",
  "basis": "product documentation",
  "supersedes": null
}

Not because the LLM necessarily needs every property. Because every downstream agent now has the opportunity to preserve provenance. That attacks #8 directly and helps #5, #9, #11 and #12 indirectly.

And the gap here is close to total (corpus scan). 73 sites publish an A2A agent card and 15 of them are valid; 58 are missing fields the card is required to carry. Of the 191 sites publishing MCP authorization metadata, 121 are valid. The machinery for machine-readable, attributable answers exists, it's barely deployed, and where it is deployed it's usually wrong.

Where I'd be much more skeptical

Base model correction

This is where a lot of GEO marketing is BS. There's no good evidence that adding llms.txt, rewriting a page a certain way, adding some schema, or mentioning a fact five times reliably causes frontier model parametric knowledge to change.

A company doesn't control when the next model trains, its corpus, deduplication, weighting, filtering, synthetic-data pipeline or post-training. Allowing GPTBot means content may be used in training. It absolutely does not mean a factual correction will be absorbed.

Worth noting what the most-adopted intervention actually is (corpus scan): 13,730 sites publish llms.txt, making it the most widely adopted agent-facing file on the web after robots.txt and the sitemap. Ahrefs found 97% of the llms.txt files they tracked receive zero AI requests. The most popular lever is the one with the least
demonstrated demand behind it.

And current GEO evidence supports that skepticism

There's now quite a lot of GEO research. Controlled studies show altering page structure and content can increase citation visibility. But a review of 45 GEO studies concluded that the strongest evidence concerns what happens once content is already available to the generative system, and found no demonstrated stable, longitudinal, cross-platform causal method for improving organic discoverability.

That's an enormous distinction. That's how I'd communicate it to customers.

So I think there is a much better business category hiding here

Not Generative Engine Optimization, and not make AI mention your brand more. Rather:

AI factual integrity for organizations.

The product asks models hundreds of questions about the organization. For every claim it identifies, it classifies the failure mode against the twelve above. And crucially it doesn't merely give a brand score. It says this error is addressable, and here is why:

"Three of four models say Lumar cannot crawl JavaScript. The claim is wrong. There is no canonical page > stating it plainly; the fact appears once, in the eleventh paragraph of a product page, alongside two > older documentation pages that contradict it. This is failure mode 2 and 3. Publish a canonical answer > page, retire the contradictions, and re-measure."

That is empirically testable. After the change, rerun the same experiment. Not once, because these systems are stochastic, but enough times to separate a real shift from sampling noise.

Now you're not selling snake-oil “AI visibility.” You are running something much closer to observability and remediation for how agents perceive an organization.

Diagnose all twelve. Sell remediation aggressively for 2, 3, 4, 5, 7 and 8, where the organization genuinely has meaningful control. For 1, 6, 9, 10, 11 and 12, report the problem and distinguish what can be mitigated from what belongs to the model or agent provider.

That distinction may be the strongest thing about the product, because it avoids pretending every wrong answer can be fixed with another llms.txt file.

And it has a precedent in how we already report. In the same corpus scan, the one security-impact check the scanner carries, whether a site has published private key material in its public signing-key directory, fired zero times across 59 publishers (corpus scan). The honest way to report that isn't "the web is secure." It is: nobody in this sample did the dangerous thing, the sample is 59, and by the rule of three that's consistent with a true rate anywhere up to about 5%. A product that reports a null result that carefully is a product whose positive findings you can believe.

I scanned 150,074 sites. The most common mistake is from 2005

Ali Habibzadeh — Wed, 23 Sep 2026 13:55:34 +0000

Last time I wrote about building a scanner that refuses to score you for things you have not published. Plenty of people asked the obvious next question: fine, so what does the web actually look like when you measure it that way?

So I pointed it at the top of the Tranco list and let it run. Between 17 and 19 September it fetched the agent facing files and endpoints of 150,074 domains. Not a sample I picked. The list in rank order, until I ran out.

Here is what came back.

Rules for AI outnumber things for AI, eleven to one

Of the 150,074 hosts, 88,484 published something a machine can read. Everything below is counted over those 88,484 and never over the whole list, because a host that never answered is not evidence of anything.

34,160 have written a rule about AI crawlers, or published a signal about how their content may be used.
3,007 have published an interface an agent can call.

I want to be careful about that first number, because it is easy to read too much into it. It counts sites that named a crawler in robots.txt, or published a content signal, an ai.txt, or a mining reservation. It does not mean they blocked anything. A group headed by a crawler's name is just as valid with an Allow in it as with a Disallow, and from outside I cannot tell which.

So the honest version is: a third of the web that publishes anything has decided to have a policy about AI. Very few of them have built anything for it yet.

The thing I did not expect

I built this to measure new protocols. The most common fault it found is twenty one years old.

61,587 hosts publish an XML sitemap. 21,448 of them break one of the format's own rules, which is about 35%. The sitemap spec was agreed in 2005.

One fault accounts for roughly half of every problem the scanner found anywhere, across every technology it checks:

https://example.com/products/sitemap.xml

<url><loc>https://example.com/about</loc></url>
                              ^^^^^^
     the file sits in /products/, so this URL is outside its scope

14,176 sites do this. That is 23% of everyone with a sitemap.

sitemaps.org says a sitemap may only list URLs at or below its own folder. Google reads those URLs anyway if you submitted the file through Search Console, which is why so many people have never seen it break. Submit the same file any other way, or hand it to anything that is not Google, and the out of scope entries are dropped.

The rest:

Fault	Sites	Share of sitemap publishers
Lists URLs above its own folder	14,176	23%
Lists URLs on another domain	6,106	9.9%
Missing the XML namespace	2,077	3.4%
Is not a sitemap at all	1,562	2.5%
Uses relative URLs	492	0.8%

Worth splitting by how much it costs you. 11,780 sites break only the folder rule, which Google forgives if you submitted it there. 9,668 are wrong no matter how you submit them. That is one in six sites with a sitemap.

If you take one thing from this post: go and look at your sitemap before you think about anything with "agent" in the name.

Which crawlers people actually name

33,626 sites name at least one crawler in robots.txt. The ones they pick:

Token	Sites naming it	Share
`GPTBot`	10,509	31%
`ClaudeBot`	9,038	27%
`CCBot`	7,929	24%
`Google-Extended`	7,280	22%
`ChatGPT-User`	6,712	20%
`Bytespider`	6,606	20%
`Amazonbot`	6,241	19%
`PerplexityBot`	5,979	18%

CCBot at 24% is the one that gives the game away. Common Crawl has been around for over a decade and has nothing to do with the current wave. Seeing it that high, in that company, tells you a lot of these rules were pasted from a list someone found rather than decided by the site.

What the three per cent built

The 3,007 sites with something callable, by what they published:

	Sites
UCP profile	2,202
OpenAPI description	497
MCP server card	368
MCP authorization metadata	191
A2A agent card	73
NLWeb endpoint	8

Eight NLWeb endpoints, across 88,484 hosts that publish anything at all.

When these do break, they break in boring ways rather than dramatic ones. 176 of the 497 OpenAPI descriptions describe a value without saying what type it is. 112 leave an operation with no operationId, so an agent picking a call has nothing stable to name it by. Of the 73 A2A agent cards, 54 are missing fields the spec requires.

llms.txt is its own story. 13,730 sites publish one, which makes it the second most adopted thing in the whole study after the sitemap. 1,022 of those start with text before the title, which the format does not allow.

What this does not say

It is not an adoption rate for the web, and I would rather say so myself than have it quoted as one. The list behind it ranks domain names by popularity, not websites. 27,992 of these names never replied, another 14,227 turned me away, and a good chunk of the rest are not sites at all: nameservers, CDN endpoints, API hosts, parked domains.

Absence is still never counted as a fault, which is the whole argument of the last post. No MCP server card means nothing is reported. An MCP server card whose endpoint does not answer is a defect. Those two stay apart everywhere in this data.

And one caveat I would rather volunteer than be caught on. Each host was put into exactly one industry by a language model reading its home page. It placed 84,149 of the 88,484. The categories are mine, not a standard taxonomy, and I have not run an accuracy audit on them. So treat anything I say about industries as indicative, not measured.

Run it on your own site

Same checks, same rules, one command:

npx agentic-readiness example.com

It exits 0 when you have not adopted something and non zero only when something you did publish is broken. That is deliberate, and it means you can put it in CI without it failing your build over a protocol you have never heard of.

Agentic readiness of example.com

  OK    Access policies (robots.txt / ai.txt / TDMRep)
  OK    Sitemap

  2 of 14 technologies published, 0 with defects.

  Nothing published here is broken. Absence is never reported.

The full study has the per industry breakdown and every fault counted. The findings are published under CC BY 4.0, so quote them, chart them, build on them, including commercially, as long as you credit us.

Same offer as last time. Take it, argue with it, or tell me where I have got it wrong.

Building an honest scanner for the agentic web

Ali Habibzadeh — Thu, 27 Aug 2026 23:19:25 +0000

Why I started

Every few weeks somebody sends me an "AI readiness score" for a website. It is always red. It never explains what the missing items would actually do for that site, and it never distinguishes between two completely different situations:

You have not published an emerging protocol.
You have published one, and it does not work.

The first is usually an observation. The protocol may not apply to your site, its adoption may be negligible, real agents may not touch it, and waiting may be the correct decision. The second is a genuine defect: an agent card advertising an endpoint that answers nothing, a discovery document that is not discoverable, a required field that is missing.

Collapsing those two into one number is how you manufacture urgency. A low score creates panic without explaining whether anything is at stake. A high score hides real breakage, because unrelated features compensate for broken ones. Both are dishonest, and both are commercially convenient, which is presumably why they are everywhere.

There was a second motivation, closer to home. The people who want these technologies are usually SEOs and content teams. The people who can ship them are engineers with a roadmap full of platform work. So a finding sits in a backlog for two quarters, or an agency charges a lot of money for a file that takes ten minutes to write. Telling someone what is broken is only half a product. I wanted the other half.

So the goal became: measure honestly, then help implement.

What I built in the container

The core is a Lumar custom metric container (inputType: Puppeteer), organised into six categories:

Category	Technologies
Discoverability	4
Content	4
Access	6
Capabilities	7
Commerce	6
Trust	3

Thirty technologies in total. 23 are scored. 7 are watched and deliberately not scored. That split is the product, not an implementation detail.

The contract: detect, then validate, then report

Every technology emits the same shape:

<tech>Present   boolean   the detection gate
<tech>Valid     boolean   present and defect-free
<tech><Defect>  boolean   one per known defect kind, true only when present-but-wrong

The rule that falls out of this is the one the whole container rests on: if a technology is not present, the collector computes nothing and emits nothing. Not Present: false. Nothing at all. There is no point validating, storing, or reporting anything about a technology a page never adopted.

That produces 309 per-URL metric fields and 185 site-wide fields, backed by 111 catalogued defects. Every defect names the clause and the revision it comes from, and whether the authority is a standard or a Lumar recommendation. We do not borrow a specification's authority for our own preferences, and the catalogue makes that distinction machine-readable rather than a matter of tone.

Two phases, because two kinds of fact

Page-level signals (canonical, JSON-LD, server-side rendering, Markdown twins, WebMCP) vary per URL and are read in the request handler. Site-wide signals (robots.txt, ai.txt, /.well-known/*, live protocol probes) are one fact per host, so they run once per host in a postCrawl handler and emit a crawl-level item. Nine site-wide signals, one row per host per signal.

Getting this wrong is expensive. Fetch a well-known file once per URL and a 50,000-page crawl knocks on somebody's server 50,000 times for a file that has not changed.

Naming evidence states, and never faking one

This is where most of my early false positives came from, so it is now a hard rule.

A fetch outcome is a discriminated union: complete, truncated, absent, unreachable. Only complete carries content, so a body we stopped reading at a byte cap cannot be handed to an analyzer, because the type does not expose it.

And more importantly:

Only an HTTP response proves absence. A 4xx or 5xx is the server saying "not here". Everything else, our own timeout, a spent budget, a reset connection, a blocked destination, is us failing to look, and must map to that signal's "unassessed" state.

Collapsing those convicts a site of our own timeout. That is the exact false positive this container exists to avoid, and the type system now refuses to let me write it.

Two registries that answer different questions

src/defect-catalog/ says what can go wrong with a technology and on whose authority. One entry per emitted flag.
src/tech/ says what the technology is: name, summary, adoption level, named adopters, the documents it is pinned to, the fixtures that demonstrate it.

Everything public is generated from that pair. The scanner, the documentation pages, the survey tables, and the report catalogue all come out of the same two files, so they cannot disagree with the code. When I add a defect, its documentation, its report column, and its public reference page appear because they are derived, not because I remembered to update three places.

Keeping up with specs that move weekly

This is the part I did not anticipate, and the part I am most pleased with.

These are emerging standards. MCP has shipped multiple revisions. Content Signals appeared and gained real adoption inside a year. Proposals turn up with a website, a GitHub repo and an enthusiastic announcement, and some of them die quietly.

A container pinned to a spec revision goes stale in weeks. Manual re-checking does not survive contact with a roadmap. So I built three Claude Code skills, versioned in the repo alongside the code they maintain. Each one has a bounded remit and writes to a bounded set of files, which is what stops an audit quietly becoming a refactor.

`/check-correctness`: is what we score still what the spec says?

Re-reads each technology's authoritative document and compares it against what the container actually validates. It hunts four kinds of drift:

a bumped spec version
a renamed or restructured field
a new present-but-wrong surface we should be catching
a check the spec no longer justifies

It is an audit, not a rewrite. It proposes changes and updates only the _Last verified_ stamps. Confirmed drift is implemented separately through /add-metric-checker, which does the full end-to-end job: interface field, collector, e2e fixtures, SOURCES.md citation, and the expectations gate.

Keeping "find the drift" and "fix the drift" as separate skills matters. An audit that can also refactor is an audit you cannot trust the output of.

`/scan-emerging-tech`: what should we admit next?

The intake step. It horizon-scans for technologies not yet in the container, and re-tests the "landscape only" and "dropped" rows for anything that has since matured.

Every candidate goes through an admission rubric:

On-thesis. Is it part of a website's public agent-facing surface? Enterprise data-lake architecture is not.
Crawl-detectable. Can we observe it from outside, without credentials?
Present-but-wrong surface. Is there a way to adopt it and get it wrong? If not, there is nothing for us to score.
Distinct. Does it overlap something we already check?
Spec-anchored. Is there a document to cite, or only a blog post?

This rubric is why the container scores 23 things and watches 7 rather than scoring everything with a logo. It is also why some things were dropped: ACP and AP2 came out because neither presented a crawl-detectable present-but-wrong surface, and llms.txt came out for lack of adoption. Both decisions are recorded with reasoning, because "we deliberately do not score this" is a claim we should have to defend.

`/track-adoption`: who is actually using this?

Maintains an adoption ladder (L0 to L4) for every technology we score or watch, with trajectory, named adopters, depth of implementation, and dated citations. It writes to one document and nothing else.

Its evidence feeds the scan rubric's adoption gate, which closes the loop: a watched technology maturing shows up here first, then routes to the scan skill, then to the checker skill. And a scored technology whose adoption collapses triggers a drop discussion.

Today's ladder, straight from the registry, tells a useful story on its own:

L4  established   robots.txt, XML sitemap, canonical, JSON-LD, schema:Action,
                  Product schema, editorial metadata, OpenAPI, Link headers,
                  language and encoding, server-side rendering
L3  widespread    AI-bot rules, Content Signals, UCP profile
L2  adopted       MCP authorization, Markdown twin, TDMRep, Web Bot Auth
L1  emerging      MCP server card, A2A agent card, WebMCP, NLWeb
L0  negligible    ai.txt

The tall bars are two decades of web standards. The short ones are this year. A tool that treats those as equally urgent is not measuring readiness, it is selling anxiety.

Running it in the Lumar crawler

Inside a Lumar crawl, the container produces two kinds of output.

Per-URL metrics land on every page row, so you can segment and filter like any other crawl field. Site-wide items land in the crawl-level table, one row per host per signal, so a robots.txt fault appears once rather than on every one of a million URLs.

On top of that sit 33 predefined reports, and every one of them obeys the same rule:

A report's filter must AND-in <tech>Present eq true before any defect predicate.

A URL enters a report only when it actually adopted the technology. There is no report anywhere in the product that lists pages for not having something. Each report also shows every one of that technology's defect columns, ordered by severity, so no defect is invisible in the grid. That ordering is derived from the catalogue and asserted in tests, because I had eleven reports inherit the interface's declaration order, which is a parser's order, not a reader's.

Turning the checkers into a free scanner

The reports are for customers running crawls. But the argument I wanted to make is a public one, so I built a free scanner: enter a domain or a URL, get an honest answer, no account.

The important design decision is what it does not do. It does not reimplement the rules. It imports the container's own collectors and analyzers directly. The free tool and the paid audit run the same code, so they cannot drift apart and start disagreeing about what a valid MCP card looks like.

The output mirrors the container's shape: what you publish, whether each published thing is correct, and, when we could not test something properly, a plain statement that it was only partly assessed. That third state matters. A card declaring gRPC cannot be verified with an HTTPS request, so we say so instead of quietly certifying it.

There is no total score anywhere in the interface.

Builders: the other half of the product

Finding a fault and leaving you with it is the behaviour I was complaining about at the start. So every technology the scanner checks ends the same way: with something to do next.

Six builders generate a finished file: AI-crawler rules, Content Signals, ai.txt, TDMRep, an A2A agent card, an MCP server card. They interview you for what the site cannot tell us, read what you already publish so the form arrives half-complete, and generate the artifact.

Then the rule that makes it trustworthy:

Every generated artifact is run through the analyzer this container scores live sites with, and anything that fails is withheld with the reason.

We are not going to be the tool that emits the defects it reports. If a card would advertise an endpoint that does not answer, we do not write the address until the endpoint answers.

Seventeen more are tutorials, for technologies that cannot be handed over as a file because they need a running server or a template change: canonical tags, JSON-LD, Markdown twins, MCP authorization, NLWeb, WebMCP, server-side rendering. These are assembled from artifacts the repository already tests, the correct and broken fixtures from the e2e suite, and the defect catalogue's clauses. A tutorial cannot drift from what the scanner scores, because it is made of the same material.

Seven technologies we refuse to build, listed by name with the reason. "We track this proposal but do not currently recommend implementing it" is a more useful answer than a missing menu item.

One thing I got badly wrong and had to redo properly: the robots.txt builders originally emitted a fragment, while every publishing recipe said "serve this artifact at /robots.txt". Following the Next.js instructions would have replaced a site's entire robots.txt with a single line, deleting the wildcard group, every rule, every named crawler and the sitemap. A fragment also cannot express "change this crawler from Allow to Disallow", so asking to block a crawler the file already allowed returned "nothing to add".

The fix was a line-level editor that produces the complete file: your current robots.txt with the change applied, every comment and ordering choice preserved, and a list of exactly what changed. It splits a crawler out of a shared group rather than changing its neighbours, and it validates the file before and after so a fault you already had does not get blamed on us.

Eating the same food

It would be strange to publish a scanner that grades agent-readiness from a website only humans can use. So the scanner publishes everything it tells you to publish, and its own pages are scored by the container in CI.

An MCP endpoint with three tools: scan a site, search the documentation, build a file. No account, no key.
An A2A agent card and an OpenAPI contract.
WebMCP registrations in the page, so a browsing agent can complete a builder wizard while you watch.
A Markdown twin of every page, served by content negotiation and at its own address.
Our own robots.txt with named AI-crawler groups, Content Signals, ai.txt, and a TDMRep reservation that matches what the Content Signals line says rather than contradicting it.

The three MCP tools compose into the actual job, which is the point of exposing them: find what is broken, read why it matters, generate the fix, check the fix against the same rules the scan uses. An agent can do the whole loop without anyone opening the page.

This is not decoration. It caught a real defect in code I had written an hour earlier. A new page declared itself a TechArticle with no author and no datePublished. TechArticle is in the Article family, so the container's own editorial checks fired: editorialMissingAuthor and editorialMissingDatePublished, two of the defects we report on other people's sites, published on our own page.

The structural fix was a test that scores every hand-written page through the container and asserts that no defect fires, so the next page cannot ship unscored. That is what dogfooding is for. Not the marketing line, the failing test.

Where it stands

The container ships 23 scored technologies and 111 catalogued defects, with 1,426 tests. The scanner adds 607 unit tests and a browser suite. Three maintenance skills keep the specs, the intake and the adoption evidence current, on a cadence rather than on a good intention.

The position underneath all of it is simple enough to fit in a sentence: missing is not the same as broken, and a tool that cannot tell the difference is not measuring your website, it is measuring your anxiety.

If any of this is useful to you, take it, argue with it, or tell me where I have got it wrong. The reasoning is all public, including the parts where I changed my mind.