DEV Community: danielgreid

A 200 Response Does Not Prove an AI Crawler Can Read Your Site

danielgreid — Tue, 28 Jul 2026 02:44:41 +0000

A crawlability check often begins with a comforting result: request the homepage, receive HTTP 200, and declare the site open. That conclusion is too broad. The response only proves that one client, from one network, with one user agent and one request shape, reached one representation of the page at one moment.

AI crawlers do not necessarily share any of those conditions. A CDN may classify their user agents differently. A robots rule may apply to one bot but not another. A challenge page may return 200 while replacing the actual document. Even when the HTML is real, the canonical URL or robots metadata can make the page unusable for discovery.

The practical fix is to stop treating the status code as the verdict. Treat it as one item in an evidence chain.

The same URL can represent different access paths

Consider three requests to the same page. A normal browser user agent receives the product page. A generic script receives a managed challenge. A named crawler receives a 403 from a WAF rule. The URL has not changed, but the effective access policy has.

This is why a single curl command is a poor proxy for crawler access. It compresses transport, policy, identity, and content into one number. When the number is 200, it also hides soft failures: consent walls, bot challenges, login interstitials, and branded not-found pages can all be successful HTTP responses.

A useful probe records at least the requested user agent, status, final URL, content type, title, canonical URL, robots directives, and a small fingerprint of the returned body. The fingerprint does not need to retain the page. It only needs to show whether two clients received materially different documents.

async function probe(url, userAgent) {
  const response = await fetch(url, {
    redirect: "follow",
    headers: { "user-agent": userAgent }
  });

  const html = await response.text();
  const title = html.match(/<title[^>]*>([^<]*)<\/title>/i)?.[1] ?? null;

  return {
    status: response.status,
    finalUrl: response.url,
    contentType: response.headers.get("content-type"),
    title,
    bytes: Buffer.byteLength(html),
    challenge: /just a moment|verify you are human/i.test(html)
  };
}

The important property is not sophistication. It is comparability. Run the same probe with a baseline browser identity and each crawler identity, then explain the difference instead of guessing from one response.

Probe policy, transport, and page identity separately

Robots policy should be evaluated before interpreting a network response. Parse the groups for the exact bot token, apply the longest matching rule, and report which rule won. Do not assume that a wildcard group describes every named bot. Also keep robots decisions separate from firewall decisions: an allowed robots rule cannot override a CDN block, and a successful fetch does not erase a disallow rule.

Next, classify the transport outcome. DNS failure, timeout, TLS failure, 429, 403, and a Cloudflare challenge are different root causes. Lumping all of them into “blocked” makes the suggested repair unreliable. A timeout suggests availability or routing work. A 429 suggests pacing. A hard WAF block requires a rule review. A challenge means the probe did not obtain the target document, even if the status is 200.

Finally, verify page identity. Compare the final URL and canonical path with the requested page. Look for a plausible title and expected content markers. Detect obvious soft-404 titles. Read page-level robots metadata. Only after those checks should the system say that the crawler received the intended, indexable document.

I use this layered approach in a free AI crawler accessibility checker because a useful diagnostic should explain which gate failed, not merely display a red badge.

Make uncertainty an explicit result

Some sites verify crawler IP ranges or signed requests in addition to the user agent. A public diagnostic cannot honestly impersonate those network identities. In that case, the correct result is not “allowed” or “blocked.” It is “the user-agent path passed, but origin verification was not tested.”

That distinction matters operationally. False confidence can leave important pages invisible, while false alarms encourage site owners to weaken WAF rules unnecessarily. Both are worse than a bounded answer.

The same principle applies to remediation. Recommend the smallest layer-specific change and preserve the evidence that justified it. If a robots group blocks a bot, show the matching lines. If a CDN returns a hard block, capture the classification and request identifier without storing cookies. If the page is a soft 404, show the title and canonical mismatch. A future recheck can then prove whether the specific failure changed.

A status code is still useful. It is simply not a conclusion. Crawlability is the conjunction of policy permission, network reachability, document identity, and page-level indexability. Measure those gates independently, preserve the first real cause, and leave uncertainty visible when the probe cannot test a provider-specific identity.

Inferring Site Structure When the Site Doesn't Have Any

danielgreid — Sat, 25 Jul 2026 08:07:41 +0000

Most sites do not have an information hierarchy. They have a navigation menu, a sitemap a plugin generated three years ago, and a pile of pages whose relationships exist only in the head of whoever built it. If you are writing a crawler that has to emit a structured summary of a site — a table of contents, an llms.txt, a doc index — you cannot read the structure. You have to infer it.

I have written this crawler more than once now. Here is what turns out to matter.

Three structure signals, and how much to trust each

There are three sources of hierarchy on a normal website, and they disagree constantly.

The sitemap is the most complete and the least meaningful. sitemap.xml gives you every URL, but it is a flat bag, and <priority> is noise almost nobody sets deliberately. What it is good for is a denominator: if your crawl found 40 pages and the sitemap lists 900, your crawl is broken.

The nav is the most meaningful and the least complete. Whatever a human put in the header or sidebar is a curated statement of "these are the things that matter," and it gives you grouping for free — a <ul> under a heading in a sidebar is a section. The catch is that nav typically covers 5–20% of the URL space, and on marketing sites it is often pure funnel (Pricing, Book a Demo) rather than content structure.

URL path depth is the cheap fallback, and right more often than it deserves to be. /docs/api/auth/tokens really does usually mean tokens is under auth is under api. The failure mode is flat-by-design sites where everything is /p/<slug>.

My rule: nav is ground truth where it exists, URL path fills gaps, sitemap validates coverage. Never let path depth override an explicit nav grouping.

def infer_parent(url, nav_index, url_tree):
    if url in nav_index:                    # human-curated, trust it
        return nav_index[url].section

    parts = urlparse(url).path.strip("/").split("/")
    for depth in range(len(parts) - 1, 0, -1):
        candidate = "/" + "/".join(parts[:depth])
        if candidate in url_tree:           # ancestor we actually crawled
            return candidate

    return ROOT   # give up honestly rather than invent a parent

That last branch matters more than it looks. The temptation is to always produce a tidy tree, so you invent a parent from a URL segment that was never a real page — and now your output claims /blog/2024/ is a section when it 404s. A flat node attached to root is worse-looking and more correct.

SPA nav: don't render the whole site

A lot of nav is rendered client-side, so fetch() returns a shell with an empty <nav>. The obvious fix is to run everything through headless Chrome. That fix costs you roughly two orders of magnitude in throughput, and you do not need it for most pages.

The approach that has held up is a two-tier crawl with escalation. Fetch plain HTML first; escalate to a rendered fetch only when a heuristic says the static HTML is content-free:

def needs_render(html):
    text = extract_visible_text(html)
    if len(text) < 200: return True                  # shell, no content
    if count_internal_links(html) == 0: return True  # no nav at all
    if has_root_mount(html) and len(text) < 1000:
        return True   # <div id="root"></div> plus a spinner
    return False

Critically, you usually only need to render one page to recover the nav, because the nav is the same everywhere. I render the homepage, one deep page (to catch section-only sidebar nav), and anything that trips the heuristic. Three renders instead of nine hundred.

Also: check for an embedded state blob before reaching for a browser. Next.js dumps __NEXT_DATA__, Nuxt dumps __NUXT__, and plenty of frameworks inline their route manifest. Parsing JSON out of a script tag is free and gives cleaner structure than scraping rendered DOM.

Deduping near-identical URLs

This is where naive crawlers produce garbage. The same document is reachable as /docs/auth, /docs/auth/, /docs/auth/index.html, /docs/auth?utm_source=twitter, and /docs/auth#tokens — while /docs/auth?page=2 looks identical to all of those and is not a duplicate.

Cheap normalization handles the first tier: lowercase host, strip fragment and trailing slash, sort query params, drop a denylist of tracking params (utm_*, fbclid, gclid, ref). Do not strip all query params — you will collapse paginated and filtered pages carrying real distinct content.

Then honor what the site tells you. If <link rel="canonical"> points elsewhere, the owner is explicitly declaring the duplicate; follow it. Same for rel="alternate" hreflang — translations are language variants, not duplicates, and must never be flattened together.

For what survives, content-level dedup catches the rest. Full-text hashing fails on one differing footer timestamp, so hash the shape: normalized H1 + ordered H2s + rounded body length.

def structure_fingerprint(doc):
    heads = [normalize(h) for h in doc.headings if h.level <= 2]
    bucket = round(len(doc.text) / 500)   # tolerate small diffs
    return sha1("|".join(heads + [str(bucket)]))

Identical fingerprints mean pick one — prefer the shortest URL, then the one appearing in nav, then the sitemap. Deterministic tie-breaks matter: if the winner flips between runs, your diffs become useless.

The whole pipeline is roughly: sitemap for coverage → static fetch with selective render escalation → normalize and canonicalize → structure-fingerprint dedup → merge nav grouping over the path-derived tree → emit. When I finally wrapped this into something usable rather than a folder of scripts, it became aicrawlable.com, which crawls a URL and emits spec-shaped llms.txt out the other end — but the interesting part was always the inference, not the file format.

The part nobody warns you about

Hierarchy inference degrades badly and silently. A crawler that produced a beautiful nested tree last month produces a flat list this month because the site moved its nav into a client-rendered mega-menu. Nothing errors. It still exits 0.

So instrument the shape of your own output: tree depth, orphan ratio, dedup collapse rate, render escalation rate. Alert on deltas, not absolutes. A jump from 4% orphans to 60% is the signal that your nav extraction broke — and it is the only signal you get.

llms.txt: What It Actually Does, and Why It Rots

danielgreid — Fri, 24 Jul 2026 18:46:10 +0000

When ChatGPT, Claude or Perplexity answers a question about your product, it is not consulting a decade of PageRank. It fetches a handful of pages and tries to work out what your site is. That is a very different retrieval problem from classic search, and most sites are accidentally hostile to it.

Why sitemaps are the wrong mental model

A sitemap optimises for coverage — every URL, every paginated archive, every tag page. That is correct for a crawler with a huge budget and a ranking model to sort the noise afterwards.

An LLM landing on your site has neither. It has a limited context window and one shot. If the first thing it ingests is 400 URLs of ?page=17 and /tag/misc, your three genuinely useful guides are buried.

llms.txt inverts this. It is a small markdown file at your root that optimises for priority:

# Your Product

> One-line description of what this actually does.

## Docs
- [Quickstart](https://example.com/docs/quickstart): Install and first request in 5 minutes
- [API Reference](https://example.com/docs/api): Every endpoint with request/response examples

Two rules make or break it:

Every link carries a description. The colon-suffix annotation is what lets a model decide whether to fetch a page. A bare link list is barely better than a sitemap.
Omit aggressively. If a page does not answer a question someone would ask, it does not belong.

llms-full.txt and the context tradeoff

llms-full.txt inlines expanded content rather than linking out, so a model can ingest everything in one request. This is genuinely useful for compact docs — and actively harmful for large sites, where you will blow the context window and get truncated mid-document.

Rough heuristic: if your docs exceed roughly 50k tokens, ship llms.txt alone and let models fetch selectively.

The part nobody mentions: it rots

This is where most implementations quietly fail. You write the file, ship it, and three months later half the descriptions describe features you renamed and two links 404. Nothing in your build catches it, because nothing validates it.

Two things worth automating:

Link liveness — CI check that every URL in llms.txt still returns 200
Regeneration on docs change — treat it as a build artifact, not a hand-maintained file

I ended up building a free generator for exactly this loop, after my own docs got cited with a description I had rewritten months earlier. Paste a URL, it crawls the structure, and produces spec-compliant llms.txt and llms-full.txt you can edit before exporting — reorder, drop low-value pages, rewrite descriptions. No signup, nothing stored: aicrawlable.com.

Worth being honest about

No search engine has committed to llms.txt as a ranking signal, and it may never become one. What it does today is cheap insurance: when a model does fetch your site, it reads a curated map instead of guessing. That asymmetry — near-zero cost, meaningful upside — is the whole argument.