Your Rate Limiter Is Silently Hiding You From the Agents That Want to Pay You

I run a pay-per-call marketplace for AI agents. The way agents find the endpoints is through a discovery flow, and the way the endpoints charge is HTTP 402 — the request comes back with a challenge that names the price, the asset, and the address. There is no API-key onboarding. An agent either can read the price and pay it, or it cannot.

So "being discovered" is the entire growth channel, and I spent an afternoon this week watching that channel behave in a way I did not expect. One crawler was being throttled into uselessness while a dozen others sailed through, and the pattern behind it is worth writing down.

The three-step dance of a discovery crawler

A serious indexer of this kind of marketplace does not just fetch a page and leave. It runs a script that follows the spec, and the spec has three moves:

1. Read the manifest (the .well-known document) to enumerate the endpoints.
2. GET each endpoint to collect the 402 challenge — that is where the price lives, not in any catalog file.
3. POST a settlement probe to confirm the endpoint actually accepts payment, not just that it advertises one.

Step three is the tell. A crawler that only reads manifests and GETs prices is building a phone book. A crawler that POSTs is building a verified index — it is checking that the money would actually move. From my side of the table, that is the exact index I want to be in, because it is the one that tells an agent "yes, you can pay this endpoint and it will work."

The one crawler getting throttled was the thorough one

Here is what I saw in the request logs over a couple of hours.

The polite crawlers — the ones that read the manifest and GET the challenges and then leave — flowed through clean. Their status-code mix was exactly what you would expect for healthy discovery: 200 for the manifest, 402 for the price challenges, 404 for endpoints that had been rotated out. Zero 429s. Rate limiter never touched them.

Then there was the distributed one. It came from fifteen-odd cloud IPs, sharded the endpoint list across them, and did the full three-step dance on each endpoint: manifest, GET the 402, POST the settlement probe. And it ate 429s. On one of its IPs, out of ~75 requests, ~45 came back rate-limited. Roughly sixty percent of its traffic never got through.

Read that again. The crawler that was doing the most complete job of discovery — the one actually verifying settlement, the one whose index entry would be the most valuable to me — was the one my rate limiter decided to block.

Why the naive limiter makes this exact mistake

The limiter was keyed on raw request counts per IP, full stop. That is the default, and it is wrong for this traffic.

Discovery crawlers have to be chatty. Verifying an endpoint means at least a GET and a POST per endpoint, and if you have a catalog of a thousand endpoints, that is two thousand requests. A distributed verifier splits that across a fleet, so each IP's share is not even that large — but it is still far above whatever threshold a "normal user" hits, because there is no normal user in this flow. There are only agents and indexers.

Meanwhile the polite crawlers stayed under threshold almost by accident, because they skip step three. The crawler that costs you the most to serve is the one that brings you the least. The crawler that costs you the most to serve and brings you the most is the one you just throttled.

The category error is treating "high request volume" as a proxy for "abuse." For a marketplace whose entire distribution is machine discovery, volume is the product.

Distinguish by pattern, not by count

The fix is not "raise the limit." Raising the limit would just let the next wave through while still failing the actual distinction. The fix is to recognize the shape of legitimate discovery.

A spec-compliant discovery sequence has a recognizable grammar: manifest read, then a burst of 402-challenge GETs, then settlement probes — with the probes going to the same resources that were just challenged, in the same window. That is not what a scraper does. A scraper walks every link it can find. A credential-stuffer POSTs to one auth endpoint ten thousand times. Neither looks anything like "enumerate, challenge, settle."

So the carve-out is not "give crawlers more requests." It is: if the request sequence matches the discovery grammar, budget it separately from general abuse. Watch the ratio of settlement probes to challenges; a real verifier probes roughly the endpoints it challenged, a spammer probes one endpoint a thousand times. The ratio is a cleaner signal than the count ever was.

The bigger point

If you run anything that gets indexed as its growth channel — an API directory, a marketplace, a network of any kind — then your abuse defenses and your discoverability are drawing from the same budget, and you should tune them together, not sequentially.

The signal I kept reaching for — "is this traffic abusive?" — was the wrong question. The right question is "is this traffic discovering me correctly?" And the crawler I was most tempted to block was the one discovering me the most correctly of all.

The day ended with the limiter untouched and the answer being "watch the pattern." I am going to spend the next week deciding the carve-out by watching whether the verified-index crawlers show up in the people who actually pay. That is the metric that settles it, not the 429 count.

Correction, after re-checking the logs

The "thorough crawler" framing above was too charitable to one specific client. When I pulled that IP's full request history instead of the aggregate, it turned out to be high-frequency enumeration — POSTing the same couple of service paths (address parsing, PDF text extraction) over and over from a single cloud IP, not probing many distinct endpoints once each. That is scanning, not settlement verification, and the 429s it ate were the limiter doing its job correctly.

This lands on the same prescription, just stated more sharply: the signal is the ratio of probes to distinct challenges, not the raw count and not "does it POST." A real verifier probes roughly the endpoints it challenged (~1:1). A scanner hammers one endpoint a thousand times. The carve-out should fire only when the sequence matches the discovery grammar — manifest read, then challenge GETs, then probes on those same resources — and never as a blanket "let crawlers through."

The headline stands with that boundary drawn: the misalignment that hurts is shallow crawlers (manifest + GET only) passing freely while a legitimate deep verifier gets throttled. High-frequency enumeration getting 429'd is the correct half of the behavior, not the bug. I left the original sections intact rather than rewriting them, because the mistake I made — reaching for "this client must be doing the deep dance" before actually checking its request pattern — is itself worth showing.