GPTBot is hammering your API. Block it, or charge it?

gibrancorbin11-hub — Thu, 30 Jul 2026 00:49:03 +0000

Check your logs for last month. Filter user-agents for GPTBot, ClaudeBot, PerplexityBot, Google-Extended, bingbot. For most SaaS apps and content APIs in 2026 that's now a real percentage of total requests — traffic you pay to serve, from callers who never convert, never see an ad, never subscribe.

The reflex is robots.txt or a blocklist. Three problems:

Blocking is leaky. Polite bots honor robots.txt; the traffic that costs you most doesn't announce itself.
Blocking is blunt. Some agent traffic is your future distribution — an AI assistant citing your data to a user who becomes a customer.
Blocking earns $0. The traffic has value to the caller. Blocked value is just destroyed, not captured.

There's now a third option, because two standards landed:

Web Bot Auth (RFC 9421 profile): agents cryptographically sign requests; you verify against the operator's published keys. Identity stops being a user-agent string anyone can type.
HTTP 402 + x402: a machine-readable "this costs $0.002, here's how to pay" that agents can settle without a human in the loop.

Which turns your bot problem into a pricing decision per lane:

Verified, signed agents → charge per call on the routes that cost you money. They're businesses; API fees are a line item to them.
Declared-but-unsigned bots → cheap tier or tight rate limits.
Undisclosed automation → paywall or block. Sneaking shouldn't be free.
Humans → free, always. They never see any of this.

Wiring it by hand means reading RFC 9421, fetching and caching key directories, building the signature base, and handling replays. Or:

npm i wayleave

app.use(new Wayleave({
  directories: { 'https://agents.example/keys': { 'k1': pubkey } },
  pricedPaths: { '/api/expensive': 0.002 },
}).express());

One middleware line. Agents get a 402 with a price; your users notice nothing. The interesting question stops being "how do I stop bots" and becomes "what's my agent traffic worth" — and that's a much better problem to have.

What the middleware doesn't do

It prices the request. It doesn't keep your books.

Charging per crossing means knowing what you're owed: a durable record of every billable request, deduplicated, attributed to a cryptographic agent identity rather than an IP guess, and turned into something you can invoice against. The middleware emits a usage event for each priced call and hands it to you. What happens next is your problem — a queue, a table, a reconciliation job.

I'm considering building the other half: a hosted meter that receives those events, keeps the ledger, handles key directory fetching and rotation, and produces billing records per agent operator. Never touching your funds — settlement stays on a payment rail you choose. The meter is the meter, not a middleman.

It isn't built. Not a beta, not a private preview — unbuilt. Whether it gets built depends on whether people running real agent traffic actually want it, so there's a list:

→ wayleave.dev

One email if it becomes real, nothing otherwise. There's one optional question on the form — roughly how much agent traffic you see — and that answer is worth more to me than the signup.

The middleware is free and finished today either way:

npm i wayleave

MIT, zero dependencies, source at github.com/gibrancorbin11-hub/wayleave.

DEV Community: gibrancorbin11-hub

GPTBot is hammering your API. Block it, or charge it?

What the middleware doesn't do