How to Identify AI Agents When Anyone Can Fake the User-Agent

Your access log says ChatGPT. Or Claude, or Google-Agent. The line looks official, but the name in it is whatever the client chose to send, and any scraper can send the same one.

That used to be a headache for search engines and big publishers. Now AI agents compare prices and fill carts for real people, and every site with a firewall rule or a rate limit runs into it. Knowing how to identify AI agents comes down to one question: who sent this request, and can they prove it?

A User-Agent is what a bot says about itself. A signature is something it has to prove, with a key an impostor doesn't have.

Why AI agents are harder to identify than crawlers

Classic crawlers are the easy case. Googlebot crawls from Google's own addresses, and those addresses have hostnames that point back to Google. Agents break that pattern in three ways:

• They act for a person, in real time. An agent opens a page because someone asked it to, and for user-triggered fetchers like ChatGPT-User, robots.txt rules may not apply.
• They often drive a real browser. The request looks like Chrome from the TLS handshake to the last header, so the tricks that catch simple scripts see a browser.
• They run on shared infrastructure. Plenty of agents work inside cloud browsers hosted by third-party platforms, so the IP address belongs to the platform and every other customer exits from the same pool.

The old ways to detect AI agents either can't see the agent at all, or can't tell it apart from an impostor on the same network.

Four ways to check a bot's identity, and what each can fake

Method	What it checks	Can an impostor fake it?	Works for agents on shared clouds?
User-Agent string	Text the client writes about itself	Yes, in seconds	No
Published IP ranges	The address is on the operator's list	Not from outside, but co-tenants on those IPs look identical	Poorly: the list names the platform, not the agent
Reverse DNS	The IP's hostname is the operator's and resolves back to it	No	Rarely: mostly search crawlers set this up
Signed request (Web Bot Auth)	A key published on the operator's domain signed the request	Not without the private key	Yes: the key travels with the agent

The forward half of the reverse DNS check is the part that matters. Anyone can set the reverse name of their own IP to googlebot.com, but only Google can make that name resolve back to it.

The first three methods infer identity from something next to the request: a string inside it, the address behind it, a DNS record somewhere else. A signature puts the proof inside the request itself.

How signed requests work

Web Bot Auth is built on HTTP Message Signatures, RFC 9421, a finished IETF standard since February 2024. The part that tells bots how to use it is still a set of drafts in an IETF working group, so details can change. The flow:

1. The operator creates a key pair. In practice that's Ed25519, a small, fast elliptic-curve key. The private half stays with the operator.
2. It publishes the public half in a key directory: a small JSON file at a fixed address on its own domain, /.well-known/http-message-signatures-directory. The file is served over HTTPS, so the domain's TLS certificate vouches for who published those keys.
3. The agent adds three headers to every request. Signature-Agent says where its keys live. Signature-Input lists what was signed: at least the host being requested, plus a creation time, an expiry time, the key's ID and the tag web-bot-auth, which marks it as a bot signature. Signature carries the signature itself.
4. The site checks. It reads Signature-Agent, fetches the directory (once, then from cache, even for an agent it has never seen), picks the key by its ID, rebuilds the signed text from the request and verifies the signature. With the keys cached, that's one signature check per request.

Two details make this hard to abuse. The signature covers the host, so a copy captured on one site won't verify on another. And it expires. The drafts recommend a lifetime of no more than 24 hours, so a stolen copy goes stale on its own.

What a valid signature proves, and what it doesn't

A valid signature tells you two things: the request was signed by someone holding a key that the domain in Signature-Agent publishes, and it was signed inside its validity window.

It doesn't tell you:

• that the agent is harmless, or allowed to do what it's doing. Good bot vs bad bot is still your call, made on behavior;
• that the request is fresh. A copied signature can be replayed until it expires, unless the site also demands a one-time value (a nonce) and remembers the ones it has seen;
• who the human behind the agent is. The protocol doesn't authenticate users at all.

A missing signature proves nothing either. Your own browser doesn't sign requests, and neither does most legitimate software.

One rule follows from all of this. Attach your policy to the domain whose key verified the request, never to whatever the Signature-Agent header says. The header only tells you where to look for the key.

Who signs today, and who checks

Adoption is early, and it changes month to month:

• The cloud browser behind ChatGPT's agent features signs its outbound requests, with Signature-Agent set to chatgpt.com.
• Google signs some Google-Agent requests, as agent.bot.goog. The rollout is experimental and partial, so IP and reverse DNS checks stay as the fallback.
• Classic crawlers such as GPTBot are still verified by their published IP lists.
• On the checking side, Cloudflare, AWS WAF, Akamai and Vercel verify these signatures for their customers. Cloudflare accepts a valid signature as one of three ways a verified bot can identify itself, next to an IP list and reverse DNS.

How to identify AI agents on your site

1. Stop letting the User-Agent open doors on its own. A rule that waves through anything calling itself ChatGPT waves through every scraper that copies the name, and faking a User-Agent takes one line of config.
2. Check what your CDN or WAF already does. Behind one of the services above, signed agents may already be verified and labeled for you. Point your rules at that label, not at the string.
3. Log the three signature headers. Default access log formats skip them, and without them you can't check a request after the fact.
4. Keep IP lists and reverse DNS for everything that doesn't sign, and refresh the lists on a schedule.
5. Sort requests into three groups: verified (policy goes to the signing domain), signed but failed (suspicious, often just a misconfigured client) and unsigned (unknown, which is different from guilty).
6. Keep your rate limits. A verified agent hitting your search endpoint 50 times a second is still hitting it 50 times a second.

To see what a real signed request proves, paste one, headers included, into the Agent Verifier. It shows what the signature proved and what the key directory publishes, and when something fails, it names the fault and the one change that fixes it. Nothing is stored, and an expired signature works too, checked as of the moment it was made.

If you build agents: sign, then check from the outside

On rented cloud IPs, a signature is the one identity signal that doesn't depend on the neighbors. The catch is that a broken one fails silently. From the client side, a malformed signature looks exactly like an agent the site has never heard of. The usual suspects:

• signed text built one character differently from what the verifier rebuilds, like missing quotes or an extra line break;
• a key directory your own CDN refuses to serve, because a bot rule blocks the verifier;
• a key ID that's a short label instead of the key's thumbprint;
• a redirect between the bare domain and www, which verifiers following the draft won't follow.

All four look fine from your own machine. Paste a request your agent really sent into the Agent Verifier to see which one you hit, or follow the implementation guide step by step: key, directory, signature, check. Your agent can also check itself. Connect the packet.guru MCP server and it finds a self-check tool that takes no arguments and reports how its own request arrives, signed or not.