DEV Community

gibrancorbin11-hub
gibrancorbin11-hub

Posted on

GPTBot is hammering your API. Block it, or charge it?

Check your logs for last month. Filter user-agents for GPTBot, ClaudeBot, PerplexityBot, Google-Extended, bingbot. For most SaaS apps and content APIs in 2026 that's now a real percentage of total requests — traffic you pay to serve, from callers who never convert, never see an ad, never subscribe.

The reflex is robots.txt or a blocklist. Three problems:

  1. Blocking is leaky. Polite bots honor robots.txt; the traffic that costs you most doesn't announce itself.
  2. Blocking is blunt. Some agent traffic is your future distribution — an AI assistant citing your data to a user who becomes a customer.
  3. Blocking earns $0. The traffic has value to the caller. Blocked value is just destroyed, not captured.

There's now a third option, because two standards landed:

  • Web Bot Auth (RFC 9421 profile): agents cryptographically sign requests; you verify against the operator's published keys. Identity stops being a user-agent string anyone can type.
  • HTTP 402 + x402: a machine-readable "this costs $0.002, here's how to pay" that agents can settle without a human in the loop.

Which turns your bot problem into a pricing decision per lane:

  • Verified, signed agents → charge per call on the routes that cost you money. They're businesses; API fees are a line item to them.
  • Declared-but-unsigned bots → cheap tier or tight rate limits.
  • Undisclosed automation → paywall or block. Sneaking shouldn't be free.
  • Humans → free, always. They never see any of this.

Wiring it by hand means reading RFC 9421, fetching and caching key directories, building the signature base, and handling replays. Or:

npm i wayleave

app.use(new Wayleave({
  directories: { 'https://agents.example/keys': { 'k1': pubkey } },
  pricedPaths: { '/api/expensive': 0.002 },
}).express());
Enter fullscreen mode Exit fullscreen mode

One middleware line. Agents get a 402 with a price; your users notice nothing. The interesting question stops being "how do I stop bots" and becomes "what's my agent traffic worth" — and that's a much better problem to have.

What the middleware doesn't do

It prices the request. It doesn't keep your books.

Charging per crossing means knowing what you're owed: a durable record of every billable request, deduplicated, attributed to a cryptographic agent identity rather than an IP guess, and turned into something you can invoice against. The middleware emits a usage event for each priced call and hands it to you. What happens next is your problem — a queue, a table, a reconciliation job.

I'm considering building the other half: a hosted meter that receives those events, keeps the ledger, handles key directory fetching and rotation, and produces billing records per agent operator. Never touching your funds — settlement stays on a payment rail you choose. The meter is the meter, not a middleman.

It isn't built. Not a beta, not a private preview — unbuilt. Whether it gets built depends on whether people running real agent traffic actually want it, so there's a list:

wayleave.dev

One email if it becomes real, nothing otherwise. There's one optional question on the form — roughly how much agent traffic you see — and that answer is worth more to me than the signup.

The middleware is free and finished today either way:

npm i wayleave
Enter fullscreen mode Exit fullscreen mode

MIT, zero dependencies, source at github.com/gibrancorbin11-hub/wayleave.

Top comments (1)

Collapse
 
topstar_ai profile image
Luis Cruz

I found the discussion around Web Bot Auth and HTTP 402 + x402 particularly interesting, as it highlights the need for a more nuanced approach to handling bot traffic. The idea of charging verified, signed agents per call on costly routes makes sense, as it allows businesses to recoup some of the costs associated with serving these requests. I've worked on similar projects where we had to balance the need to prevent abuse with the desire to facilitate legitimate bot traffic, and I think the Wayleave middleware could be a valuable tool in this regard. Have you considered how the hosted meter would handle cases where an agent operator has multiple identities or keys, and how it would ensure accurate billing and reconciliation in such scenarios?