DEV Community

hao li
hao li

Posted on

Your API Gateway Rate Limit Can't See the Agent Looping Your Search Tool

#opensource #showdev #python

Your API Gateway Rate Limit Can't See the Agent Looping Your Search Tool

Here's the failure mode nobody's gateway catches.

An agent gets stuck. Not stuck like a crash — stuck like a loop: it calls web_search, gets results, doesn't like them, calls web_search again with a slightly different query. 10 times. 50 times. 400 times a minute. Each call is a perfectly valid HTTP request. Your API gateway sees normal traffic well under its per-key limit. Your bill disagrees.

The problem: gateway rate limits live at the HTTP layer. Agent abuse lives at the tool-call semantic layer. One HTTP request can fan out into many tool calls. One looping tool can spread its calls across many endpoints. A gateway counting bytes and paths never sees "the agent is hammering one expensive tool" — but that's exactly the unit an agent developer wants to budget.

I built the fuse that belongs at the tool-call layer.

What I built

tool-throttle — a zero-dependency Python package (stdlib only, Python ≥ 3.9) that puts a token-bucket rate limiter on MCP tool calls, per tool name. Wrap your tool dispatcher, set quotas, and over-quota calls get queued or rejected with a retry_after hint.

pip install tool-throttle
Enter fullscreen mode Exit fullscreen mode 
from tool_throttle import Throttler, ThrottlerConfig, Quota, REJECT

th = Throttler(ThrottlerConfig(
    default=Quota(calls=60, per_seconds=60),
    per_tool={
        "web_search":  Quota(calls=10, per_seconds=60),  # tight leash
        "expensive_*": Quota(calls=2, per_seconds=60),   # glob patterns
    },
    on_exceed=REJECT,  # or "queue": wait for a token, up to max_queue_wait
))

safe_search = th.wrap("web_search", mcp_client.call_tool)
safe_search("web_search", {"query": "..."})  # past quota -> ThrottledError

th.stats()  # per-tool allowed / rejected / queued / waited_seconds
Enter fullscreen mode Exit fullscreen mode

Quotas live in the language of the agent — tool names and glob patterns, with per-tool bursts — not in the language of HTTP. Config round-trips through JSON so it can sit in a file next to your agent config:

cfg = ThrottlerConfig(
    default=Quota(calls=60, per_seconds=60),
    per_tool={"web_*": Quota(calls=10, per_seconds=60, burst=3)},
    on_exceed="queue",
    max_queue_wait=30.0,
)
cfg.save("throttle.json")
Enter fullscreen mode Exit fullscreen mode

There's also a CLI for quick checks and a demo:

echo '{"tool": "web_search"}' | tool-throttle check --config throttle.json
echo '{"tool": "web_search"}' | tool-throttle acquire --config throttle.json
tool-throttle demo   # watch a looping agent get fused
Enter fullscreen mode Exit fullscreen mode

How it works

Each distinct tool name gets its own TokenBucket, sized by the first matching quota in per_tool (glob patterns, insertion order) or default. Buckets refill steadily — Quota(calls=10, per_seconds=60) sustains 10 calls/minute with a burst of up to burst (default: calls).

On quota exhaustion:

  • reject — fail fast with ThrottledError, carrying a retry_after hint so the agent can back off intelligently instead of hammering blindly.
  • queue — block until a token frees up, bounded by max_queue_wait; if the bound would be exceeded, fail fast instead. Queuing keeps bursty-but-legitimate agents flowing while still capping the rate.

Everything is thread-safe, time is injectable (the test suite runs against a fake clock), and there are no dependencies — it's threading, time, json, fnmatch, and argparse.

Why this isn't just "rate limiting, again"

Because the layer matters:

  • Gateway limits are per key/IP. An agent looping on one expensive tool looks normal at the HTTP layer while burning money at the tool layer. tool-throttle counts tool-call intent.
  • Tool granularity. You don't want the same budget for get_time and web_search. Per-tool quotas with globs (expensive_*) express what you actually fear.
  • Fuse, not wall. Queuing with a bounded wait absorbs legitimate bursts instead of erroring on the first spike — it's a fuse that trips on sustained abuse, not a wall that punishes normal use.
  • Observability. stats() tells you which tool the agent was abusing, with waited-time accounting. That's the number you want when you're debugging a runaway agent at 2am.

It complements gateway rate limiting; it doesn't replace it. The gateway protects your infrastructure. tool-throttle protects your wallet from your own agent.

Links

MIT licensed. Feedback welcome — especially war stories about agents you've caught looping.

Top comments (1)

Collapse
 
arhancanli profile image
Arhan Canli •

A per-tool bucket catches the loop that hammers one name, but a stuck agent often rotates: web_search returns something it dislikes, then fetch_url, then web_search with one word changed. Each tool stays under its own quota while the run as a whole burns the budget. A second counter keyed on the whole run (total calls, or total spend if tools have a cost weight) next to the per-tool ones covers that case.

Two cheaper signals for a pure loop: a hash of the normalized arguments, where the 5th near-identical query in a window is the tell, and the retry_after text going back into the model's context as a tool result it can read, not only as an exception the harness swallows.

One edge: buckets live in process memory, so with several workers behind a dispatcher the real limit is quota times worker count. Is the state meant to be shared, or is one throttler per agent run the intended scope?