<?xml version="1.0" encoding="UTF-8"?>
<rss version="2.0" xmlns:atom="http://www.w3.org/2005/Atom" xmlns:dc="http://purl.org/dc/elements/1.1/">
  <channel>
    <title>DEV Community: PurpleRain2025</title>
    <description>The latest articles on DEV Community by PurpleRain2025 (@purplerain2025).</description>
    <link>https://dev.to/purplerain2025</link>
    <image>
      <url>https://media2.dev.to/dynamic/image/width=90,height=90,fit=cover,gravity=auto,format=auto/https:%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Fuser%2Fprofile_image%2F4243170%2F309d6cfc-5915-4cac-a890-98905116419a.jpeg</url>
      <title>DEV Community: PurpleRain2025</title>
      <link>https://dev.to/purplerain2025</link>
    </image>
    <atom:link rel="self" type="application/rss+xml" href="https://dev.to/feed/purplerain2025"/>
    <language>en</language>
    <item>
      <title>Scraper or agent? Why the answer has to be checkable offline, per request, with RFC 9421</title>
      <dc:creator>PurpleRain2025</dc:creator>
      <pubDate>Fri, 13 Nov 2026 15:22:31 +0000</pubDate>
      <link>https://dev.to/purplerain2025/scraper-or-agent-why-the-answer-has-to-be-checkable-offline-per-request-with-rfc-9421-4a9a</link>
      <guid>https://dev.to/purplerain2025/scraper-or-agent-why-the-answer-has-to-be-checkable-offline-per-request-with-rfc-9421-4a9a</guid>
      <description>&lt;h1&gt;Scraper or agent? Why the answer has to be checkable offline, per request, with RFC 9421&lt;/h1&gt;

&lt;p&gt;Every site operator now has the same log file. Requests that say they are &lt;code&gt;GPTBot&lt;/code&gt;, &lt;code&gt;ClaudeBot&lt;/code&gt; or &lt;code&gt;some-shopping-agent/1.0&lt;/code&gt;, from cloud IP ranges, at rates that range from polite to hostile. Some of them are scrapers wearing a borrowed name. Some are real agents doing a job a real person asked for: booking a table, comparing prices, filing a support ticket.&lt;/p&gt;

&lt;p&gt;The site has to decide in a few milliseconds, on every request, which is which. And "block all bots" stopped being an answer the day users started sending agents to do things for them.&lt;/p&gt;

&lt;p&gt;This post goes through why the usual signals fail, what a signal has to look like to be worth anything, and what we learned building one. We ended up with three properties we think are &lt;strong&gt;non-negotiable&lt;/strong&gt;:&lt;/p&gt;

&lt;ol&gt;
    &lt;li&gt;
&lt;strong&gt;A per-request cryptographic signature&lt;/strong&gt; (RFC 9421, HTTP Message Signatures), not a token.&lt;/li&gt;
    &lt;li&gt;
&lt;strong&gt;Verification that works offline&lt;/strong&gt;, with no call to anyone per request.&lt;/li&gt;
    &lt;li&gt;
&lt;strong&gt;A registry that never sees the traffic&lt;/strong&gt; it vouches for.&lt;/li&gt;
&lt;/ol&gt;

&lt;p&gt;Our implementation is called &lt;strong&gt;AIDR&lt;/strong&gt;. It is open source on the verifier side, and you can watch the whole handshake run in your browser at &lt;strong&gt;&lt;a href="https://aidr.evolivcore.com/demo" rel="noopener noreferrer"&gt;aidr.evolivcore.com/demo&lt;/a&gt;&lt;/strong&gt; (it runs locally with WebCrypto; nothing is sent anywhere). But the argument holds for any design, so let's start with the argument.&lt;/p&gt;





&lt;h2&gt;Act I: Why the usual signals fail&lt;/h2&gt;

&lt;pre&gt;&lt;code&gt;[ Traditional Security Signals ]
              |
      "Trust the network."
              |
      -----------------
      ▼               ▼
  User-Agent      IP Allowlist
  Easily faked    Fails on shared cloud
&lt;/code&gt;&lt;/pre&gt;

&lt;p&gt;&lt;strong&gt;User-Agent strings&lt;/strong&gt; are self-reported. Anyone can type &lt;code&gt;GPTBot&lt;/code&gt;. This is not a theoretical weakness; it is the first thing a scraper changes.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;IP allowlists and reverse DNS&lt;/strong&gt; prove which network a request came from, not who sent it. They work for a handful of big crawlers with dedicated ranges. They do not work for the long tail of agents that run on shared clouds, serverless platforms and laptops. One abusive tenant on a shared range poisons it for everyone.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;API keys and bearer tokens&lt;/strong&gt; prove that someone once had the key. A bearer token copied from a log works for whoever holds it, until it expires. And a key is issued per site, so an agent visiting a thousand sites needs a thousand sign-ups.&lt;/p&gt;

&lt;p&gt;What all of these lack is the same thing: &lt;strong&gt;proof of possession of a key, bound to an identity someone is accountable for, checked on the request itself.&lt;/strong&gt;&lt;/p&gt;





&lt;h2&gt;Act II: Property 1 - Sign every request (RFC 9421)&lt;/h2&gt;

&lt;p&gt;HTTP Message Signatures (RFC 9421) let a client sign selected parts of a request. The verifier rebuilds the same "signature base" from what it received and checks the signature. &lt;/p&gt;

&lt;p&gt;A signed request in the IETF Web Bot Auth profile looks like this:&lt;/p&gt;

&lt;pre&gt;&lt;code&gt;GET /api/products HTTP/1.1
Host: shop.example
Signature-Agent: "https://agents.acme.example"
Signature-Input: wba=("@authority" "@method" "@path" "signature-agent");created=1761480000;expires=1761480300;keyid="poqkLGiymh_W0uP6PZFw-dvez3QJT5SolqXBCW38r0U";alg="ed25519";nonce="…";tag="web-bot-auth"
Signature: wba=:dGhpcyBpcyBub3QgYSByZWFsIHNpZ25hdHVyZQ…:&lt;/code&gt;&lt;/pre&gt;

&lt;p&gt;What this buys you, compared with a token:&lt;/p&gt;

&lt;ul&gt;
    &lt;li&gt;🔒 &lt;strong&gt;Bound to this request.&lt;/strong&gt; The signature covers the host, method and path. Replaying it against another endpoint fails.&lt;/li&gt;
    &lt;li&gt;⏱️ &lt;strong&gt;Bound in time.&lt;/strong&gt; &lt;code&gt;created&lt;/code&gt; and &lt;code&gt;expires&lt;/code&gt; give a short window (we use five minutes).&lt;/li&gt;
    &lt;li&gt;🛡️ &lt;strong&gt;Nothing secret travels.&lt;/strong&gt; The private key never leaves the agent. A captured request is useless on any other request.&lt;/li&gt;
&lt;/ul&gt;





&lt;h2&gt;Act III: Property 2 - Verify Offline&lt;/h2&gt;

&lt;p&gt;The obvious way to add identity is an online check: on every request, ask some registry "is this agent good?". It is also the wrong way due to &lt;strong&gt;Latency&lt;/strong&gt;, &lt;strong&gt;Cost&lt;/strong&gt;, and &lt;strong&gt;Privacy&lt;/strong&gt;.&lt;/p&gt;

&lt;p&gt;So in AIDR, the agent carries its identity with it. The verifier does everything locally. The checks run cheapest-first, so a garbage request is rejected before any cryptography runs. &lt;/p&gt;

&lt;p&gt;In Python, the whole integration is beautifully simple:&lt;/p&gt;

&lt;pre&gt;&lt;code&gt;from fastapi import FastAPI, Request
from aidr_verifier import AidrMiddleware, RelyingPartyConfig

app = FastAPI()
app.add_middleware(AidrMiddleware, config=RelyingPartyConfig(
    issuer="https://api.evolivcore.com",
    public_origin="https://shop.example"))

@app.get("/products")
async def products(request: Request):
    # "anonymous", "verified", or refused before this line
    identity = request.state.aidr          
    return {"status": identity.status, "aid": identity.aid}&lt;/code&gt;&lt;/pre&gt;

&lt;p&gt;&lt;em&gt;(Available for Flask, Django, Express, Go, Cloudflare Workers, Kong, and WordPress).&lt;/em&gt;&lt;/p&gt;





&lt;h2&gt;The Accountability Layer: A Quick Comparison&lt;/h2&gt;

&lt;p&gt;Web Bot Auth answers &lt;em&gt;which key signed this&lt;/em&gt;. Bot lists answer &lt;em&gt;which company runs this crawler&lt;/em&gt;. Neither answers the question a bank or API provider actually has: &lt;strong&gt;which accountable organisation is behind this specific agent, and can it be switched off right now?&lt;/strong&gt;&lt;/p&gt;

&lt;p&gt;Here is how the layer AIDR adds on top of Web Bot Auth stacks up side-by-side:&lt;/p&gt;

&lt;div class="table-wrapper-paragraph"&gt;&lt;table&gt;
    &lt;thead&gt;
        &lt;tr&gt;
            &lt;th&gt;Feature&lt;/th&gt;
            &lt;th&gt;Web Bot Auth alone&lt;/th&gt;
            &lt;th&gt;With AIDR on top&lt;/th&gt;
        &lt;/tr&gt;
    &lt;/thead&gt;
    &lt;tbody&gt;
        &lt;tr&gt;
            &lt;td&gt;&lt;strong&gt;Signature&lt;/strong&gt;&lt;/td&gt;
            &lt;td&gt;RFC 9421, &lt;code&gt;Ed25519&lt;/code&gt;
&lt;/td&gt;
            &lt;td&gt;The same signature&lt;/td&gt;
        &lt;/tr&gt;
        &lt;tr&gt;
            &lt;td&gt;&lt;strong&gt;Identity&lt;/strong&gt;&lt;/td&gt;
            &lt;td&gt;❓ Whoever hosts the directory&lt;/td&gt;
            &lt;td&gt;🏢 An organisation with a verified domain&lt;/td&gt;
        &lt;/tr&gt;
        &lt;tr&gt;
            &lt;td&gt;&lt;strong&gt;Granularity&lt;/strong&gt;&lt;/td&gt;
            &lt;td&gt;Usually per platform&lt;/td&gt;
            &lt;td&gt;🤖 Per agent, each with its own key&lt;/td&gt;
        &lt;/tr&gt;
        &lt;tr&gt;
            &lt;td&gt;&lt;strong&gt;Revocation&lt;/strong&gt;&lt;/td&gt;
            &lt;td&gt;Remove the key; caches decide&lt;/td&gt;
            &lt;td&gt;🛑 Signed feed, ~2 min, fail-closed at 11 min&lt;/td&gt;
        &lt;/tr&gt;
        &lt;tr&gt;
            &lt;td&gt;&lt;strong&gt;Verification&lt;/strong&gt;&lt;/td&gt;
            &lt;td&gt;☁️ Edge fetches the directory&lt;/td&gt;
            &lt;td&gt;⚡ Offline, in your own middleware&lt;/td&gt;
        &lt;/tr&gt;
    &lt;/tbody&gt;
&lt;/table&gt;&lt;/div&gt;

&lt;p&gt;Verification is free and always will be; the code that runs on your servers is Apache-2.0.&lt;/p&gt;





&lt;h2&gt;Try it yourself&lt;/h2&gt;

&lt;ul&gt;
    &lt;li&gt;
&lt;strong&gt;See the handshake:&lt;/strong&gt; &lt;a href="https://aidr.evolivcore.com/demo" rel="noopener noreferrer"&gt;aidr.evolivcore.com/demo&lt;/a&gt;. Keys are generated and checked in your browser.&lt;/li&gt;
    &lt;li&gt;
&lt;strong&gt;Read the code:&lt;/strong&gt; &lt;a href="https://github.com/EvolivCore/aiaf-sdk" rel="noopener noreferrer"&gt;github.com/EvolivCore/aiaf-sdk&lt;/a&gt;: verifiers, agent SDKs, and the shared test vectors.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;If you run a site that is drowning in "agents", or you build agents and are tired of getting blocked as a scraper, I would like to hear what breaks for you. The comments are open.&lt;/p&gt;

</description>
      <category>security</category>
      <category>ai</category>
      <category>cryptography</category>
      <category>webdev</category>
    </item>
  </channel>
</rss>
