DEV Community

denyn1
denyn1

Posted on

AI crawlers ignore robots.txt, so I built signed content permissions

#webdev #security #ai #opensource

I fell down this rabbit hole while reading why publishers keep complaining about AI crawlers. One vendor's data: 1.9 billion crawls ignored robots.txt rules in a six-month window. Another measured a 70,900:1 crawl-to-referral ratio for one major AI provider. AI bots averaged 4.2% of all HTML requests in 2025, peaking at 6.4%.

The rules are right there in a text file. The file isn't the problem. It's that a text file can't prove who wrote it, can't be revoked, and isn't bound to the domain it claims to speak for.

So I built AIFeed: an open, signed way for a site to declare what AI agents may do with its content. Here's what it does, what I measured, and what's still missing.

Three gaps plain text can't close

No proof. Anyone can copy a robots.txt to another domain, or edit the one on a compromised host. An agent has no way to check whether the rules it just read actually came from the owner.

No revocation. Once a crawler caches your "allowed for training" file, changing it later does nothing. There is no revocation list and no status to re-check on reuse.

No binding. robots.txt has no notion of "this domain, this key, this revision". Nothing chains the domain to the permissions.

I wanted something an agent could verify on its own, without calling a service and without trusting a new intermediary.

How it works

A publisher generates an Ed25519 key pair and publishes a manifest at /.well-known/ai.json. The manifest lists per-use permissions (training, retrieval, quoting, summarization), crawl rules, licensing, and a revision number. It's signed over JCS-canonical JSON, and the public key is anchored in a DNS TXT record (_aifeed).

An agent verifying a site walks that chain: TLS → domain → signature → DNS anchor. Then it re-checks a multi-signature revocation registry on every use. If the key was rotated or revoked, verification says so.

For delivery there are two profiles. AIFeed Markdown is the native one (.aifeed.md, text/aifeed+markdown) with a permission block in-band. MAKO is supported as a compatibility profile, so the same tooling can serve both. There's a signed delta index too, so an agent that already has unchanged pages doesn't refetch them.

All of it is zero-dependency. The reference implementation is Node standard library; the independent Python verifier is stdlib-only. No accounts, no SaaS in the middle.

What I measured

From the local benchmark harness (60-page corpus, 100 tenants, 4 client profiles):

  • −68.83% transferred bytes serving markdown profiles instead of HTML
  • −95.73% when a client uses the delta index and only 10% of pages changed
  • −55.19% origin bytes and −56.23% origin CPU on the publisher side
  • 0.70 ms to verify one page's signature

Caveat, and I'll keep repeating it: this is a simulation harness, not a field trial. The 30-day production pilot hasn't run. The vendor claims of 90%+ token savings you may have seen require semantic summarization, which AIFeed deliberately does not do automatically.

Try it

Publisher side, one command:

npx aifeed init --domain example.com --dir ./site
Enter fullscreen mode Exit fullscreen mode

Or sign an existing static build:

npx aifeed keygen --out .aifeed
npx aifeed site build ./public --domain example.com \
  --key .aifeed/aifeed-private.pem --llms --inject
npx aifeed validate ./public --domain example.com
Enter fullscreen mode Exit fullscreen mode

Agent side, three lines:

const { verifyRemote } = require('@aifeed/verify');
const out = await verifyRemote('example.com');
console.log(out.result, out.anchor.status); // VERIFIED anchored
Enter fullscreen mode Exit fullscreen mode

There's also an MCP server (npx -y aifeed-mcp-server) with tools for verifying manifests, fetching token-budgeted markdown, listing assets and verifying their digests. It's listed in the official MCP registry and on Smithery.

Eight live demo origins are up at https://aifeed.md if you want to poke at real signed manifests: a news site, a docs site, a store, and more.

What's still missing

  • No external cryptographic review yet, and the specs are a draft, not frozen.
  • Origin + DNS compromise is undetectable on first contact. A signature proves the key signed the file, not that the key was safe.
  • Legal review of the permission semantics is pending; the paper labels those claims as needing review.
  • The 30-day pilot hasn't happened. If you run a site with real crawler traffic, I'd like to talk.

Where to look

If you try it and something breaks, open an issue. There are a few good first issues labeled if you want to help. And if you verify your own domain and get "no manifest found", that's not a bug. Most of the web is still unsigned.

Top comments (0)