On Saturday, 17 October, Instagram suspended the No Kings Coalition's account for about 90 minutes. On Sunday, Meta's communications VP Andy Stone said a tool linked to Anthropic's Claude had accessed the coalition's profiles, posts and hashtags at roughly four times normal human speed, in short intense bursts that resembled automated bot traffic. Meta called the suspension an error and apologised. It did not say which tool it was, who ran it, or what it was doing.
So we don't know the full story, and I'm not going to pretend we do. But the mechanism Meta described is one I see constantly in agent projects: a model in a loop, a browser or HTTP tool, and nothing between them that understands "slow down". The platform's abuse systems don't care that the traffic came from a helpful assistant. They see bursts, and they act on the account or the IP behind them.
This post is the guardrail layer I now drop into every agent that touches the web. It is 124 lines of standard-library Python. I ran it against a mock server that pushes back, and the real output is below.
What "polite" means in code
Seven things, each small:
- Per-host token bucket. A steady pace per site (default 12 requests a minute, burst of 2), so the agent can't fire 40 requests the moment the model decides to "check all profiles".
- Random jitter. A human doesn't click every 5.000 seconds. A fixed interval is itself a bot signature.
-
Backoff on 429 and 503. If the server sends
Retry-After, honour it exactly, in either of its two formats (seconds or an HTTP date). If not, back off exponentially with jitter. -
robots.txt. Checked with
urllib.robotparserbefore any request, andCrawl-delaylowers the bucket rate if the site asks for it. - An honest User-Agent. Name the bot and give a contact URL and email. Site operators can then reach you instead of blocking a whole range.
- A daily action cap. A hard ceiling that a runaway loop cannot get past. Retries count, because the server sees them too.
- An append-only action log. One JSON line per action, including the ones that were refused. When something goes wrong you can answer "what did the agent actually do?" in seconds.
The module
"""polite_agent.py: keep an AI agent's HTTP traffic slow, honest and auditable.
Standard library only. Per-host token bucket, jitter, backoff that honours
Retry-After, robots.txt, an identifying User-Agent, a daily cap and a JSONL log.
"""
import email.utils, json, random, threading, time, urllib.error, urllib.request
from datetime import datetime, timezone
from urllib.parse import urlsplit
from urllib.robotparser import RobotFileParser
class Blocked(Exception):
"""Raised when a request is refused before it is sent (robots or cap)."""
class TokenBucket:
def __init__(self, rate_per_min, burst):
self.rate, self.capacity = rate_per_min / 60.0, burst
self.tokens, self.last = float(burst), time.monotonic()
self.lock = threading.Lock()
def take(self):
with self.lock:
now = time.monotonic()
self.tokens = min(self.capacity, self.tokens + (now - self.last) * self.rate)
self.last = now
wait = 0.0 if self.tokens >= 1 else (1 - self.tokens) / self.rate
self.tokens -= 1 # may go negative: the debt is paid by sleeping
if wait:
time.sleep(wait)
def utc_day():
return datetime.now(timezone.utc).date().isoformat()
def retry_after_seconds(value):
"""Retry-After is either delta-seconds or an HTTP date."""
if not value:
return None
if value.strip().isdigit():
return float(value)
try:
when = email.utils.parsedate_to_datetime(value)
return max(0.0, (when - datetime.now(timezone.utc)).total_seconds())
except (TypeError, ValueError):
return None
class PoliteClient:
def __init__(self, user_agent, log_path="agent_actions.jsonl", rate_per_min=12,
burst=2, jitter=(0.5, 2.0), daily_cap=500, max_retries=4,
base_backoff=2.0, max_backoff=120.0, timeout=20):
self.ua, self.log_path, self.timeout = user_agent, log_path, timeout
self.rate_per_min, self.burst, self.jitter = rate_per_min, burst, jitter
self.daily_cap, self.max_retries = daily_cap, max_retries
self.base_backoff, self.max_backoff = base_backoff, max_backoff
self.buckets, self.robots = {}, {}
self.today = utc_day()
self.count = self._count_today()
def _count_today(self): # the cap survives restarts because it reads the log
try:
with open(self.log_path) as f:
events = [json.loads(line) for line in f if line.strip()]
return sum(1 for e in events if e.get("day") == self.today and e.get("sent"))
except FileNotFoundError:
return 0
def _log(self, **event):
event.update(ts=datetime.now(timezone.utc).isoformat(timespec="seconds"), day=self.today)
with open(self.log_path, "a") as f: # append-only, one JSON object per line
f.write(json.dumps(event) + "\n")
def _robots_for(self, url):
parts = urlsplit(url)
origin = f"{parts.scheme}://{parts.netloc}"
if origin not in self.robots:
rp = RobotFileParser()
try:
req = urllib.request.Request(origin + "/robots.txt", headers={"User-Agent": self.ua})
with urllib.request.urlopen(req, timeout=self.timeout) as r:
rp.parse(r.read().decode("utf-8", "replace").splitlines())
except urllib.error.HTTPError as e:
rp.disallow_all = e.code in (401, 403) # 404 means no rules
except OSError:
rp.disallow_all = True # unreachable: be conservative
self.robots[origin] = rp
delay = rp.crawl_delay(self.ua)
rate = min(self.rate_per_min, 60.0 / float(delay)) if delay else self.rate_per_min
self.buckets[parts.netloc] = TokenBucket(rate, self.burst)
return self.robots[origin]
def get(self, url):
if utc_day() != self.today: # new day, fresh budget
self.today, self.count = utc_day(), 0
if not self._robots_for(url).can_fetch(self.ua, url):
self._log(url=url, sent=False, reason="robots.txt disallow")
raise Blocked(f"robots.txt disallows {url}")
host = urlsplit(url).netloc
for attempt in range(self.max_retries + 1):
if self.count >= self.daily_cap:
self._log(url=url, sent=False, reason="daily cap reached")
raise Blocked(f"daily cap of {self.daily_cap} reached")
self.buckets[host].take()
time.sleep(random.uniform(*self.jitter))
self.count += 1
req = urllib.request.Request(url, headers={"User-Agent": self.ua})
try:
with urllib.request.urlopen(req, timeout=self.timeout) as r:
body = r.read()
self._log(url=url, sent=True, status=r.status, attempt=attempt)
return body
except urllib.error.HTTPError as e:
self._log(url=url, sent=True, status=e.code, attempt=attempt)
if e.code not in (429, 503) or attempt == self.max_retries:
raise
wait = retry_after_seconds(e.headers.get("Retry-After"))
if wait is None: # no hint: exponential backoff with jitter
wait = min(self.max_backoff, self.base_backoff * 2 ** attempt * random.uniform(0.8, 1.2))
elif wait > self.max_backoff: # never retry earlier than asked: give up
raise
print(f" {e.code} on {url}, backing off {wait:.1f}s")
time.sleep(wait)
A few decisions worth explaining:
- The bucket can go into debt. If two threads grab a token at once, the second one sleeps longer. That keeps the lock short and the pace correct.
-
If the server asks for a longer wait than
max_backoff, the client gives up rather than retrying early. Retrying beforeRetry-Afterexpires is exactly the behaviour that turns a rate limit into a ban. - The cap is rebuilt from the log on startup. My first version kept the counter in memory, which means a crash loop resets it. Reading the log makes the cap real.
- If robots.txt can't be fetched at all, everything is disallowed. A 404 means "no rules", but a network failure means "I don't know", and I'd rather the agent stop.
Testing also caught a bug worth sharing. I originally wrote self.today, self.count = utc_day(), self._count_today() in __init__. Python evaluates the whole right side first, so _count_today() ran before self.today existed and crashed on a restart with an existing log. The first demo run passed because the log was empty. Test the restart path, not just the happy path.
The demo: a server that pushes back
The mock server serves a robots.txt that disallows /private and sets Crawl-delay: 1. Every third request to /profile/* gets a 429 with Retry-After: 2, and the first request to /busy gets a 503 with no hint. I set the client to 120 requests a minute so you can watch the crawl delay pull it down to one a second, and a daily cap of 8 so it runs out quickly.
"""demo.py: run PoliteClient against a local mock server that pushes back."""
import os, threading, time
from http.server import BaseHTTPRequestHandler, ThreadingHTTPServer
from polite_agent import Blocked, PoliteClient
HITS = {"profile": 0, "busy": 0}
class Mock(BaseHTTPRequestHandler):
def do_GET(self):
if self.path == "/robots.txt":
return self.reply(200, b"User-agent: *\nDisallow: /private\nCrawl-delay: 1\n")
if self.path.startswith("/profile/"):
HITS["profile"] += 1
if HITS["profile"] % 3 == 0: # every third hit: slow down, try in 2s
return self.reply(429, b"slow down", {"Retry-After": "2"})
if self.path == "/busy":
HITS["busy"] += 1
if HITS["busy"] == 1: # 503 with no Retry-After header
return self.reply(503, b"busy")
self.reply(200, f"ok {self.path} ua={self.headers['User-Agent']}".encode())
def reply(self, code, body, headers=None):
self.send_response(code)
for k, v in (headers or {}).items():
self.send_header(k, v)
self.end_headers()
self.wfile.write(body)
def log_message(self, *args):
pass
server = ThreadingHTTPServer(("127.0.0.1", 8765), Mock)
threading.Thread(target=server.serve_forever, daemon=True).start()
if os.path.exists("demo_log.jsonl"):
os.remove("demo_log.jsonl")
agent = PoliteClient(
user_agent="ResearchBot/1.0 (+https://example.com/bot; bot@example.com)",
log_path="demo_log.jsonl", rate_per_min=120, burst=1,
jitter=(0.2, 0.6), daily_cap=8, base_backoff=1.0)
start = time.monotonic()
urls = [f"http://127.0.0.1:8765/profile/{i}" for i in range(1, 5)]
urls += ["http://127.0.0.1:8765/private/inbox", "http://127.0.0.1:8765/busy",
"http://127.0.0.1:8765/profile/5"]
for url in urls:
try:
body = agent.get(url).decode()
print(f"[{time.monotonic() - start:5.1f}s] 200 {body}")
except Blocked as e:
print(f"[{time.monotonic() - start:5.1f}s] BLOCKED {e}")
print(f"requests sent today: {agent.count}/{agent.daily_cap}")
server.shutdown()
Real output from python3 demo.py on Python 3.13:
[ 0.6s] 200 ok /profile/1 ua=ResearchBot/1.0 (+https://example.com/bot; bot@example.com)
[ 1.4s] 200 ok /profile/2 ua=ResearchBot/1.0 (+https://example.com/bot; bot@example.com)
429 on http://127.0.0.1:8765/profile/3, backing off 2.0s
[ 4.6s] 200 ok /profile/3 ua=ResearchBot/1.0 (+https://example.com/bot; bot@example.com)
[ 5.8s] 200 ok /profile/4 ua=ResearchBot/1.0 (+https://example.com/bot; bot@example.com)
[ 5.8s] BLOCKED robots.txt disallows http://127.0.0.1:8765/private/inbox
503 on http://127.0.0.1:8765/busy, backing off 1.1s
[ 8.6s] 200 ok /busy ua=ResearchBot/1.0 (+https://example.com/bot; bot@example.com)
429 on http://127.0.0.1:8765/profile/5, backing off 2.0s
[ 11.5s] BLOCKED daily cap of 8 reached
requests sent today: 8/8
Read it top to bottom and every rule shows up. Requests are spaced about a second apart, not 0.5 seconds, because Crawl-delay overrode my setting. The 429 waited exactly the 2 seconds the server asked for. The 503 with no header got a jittered 1.1 second backoff. The /private URL never left the machine. And the last profile request hit the cap halfway through its retry, because retries are real requests.
The log keeps the refusals too:
{"url": "http://127.0.0.1:8765/profile/3", "sent": true, "status": 429, "attempt": 0, "ts": "2026-10-19T12:09:22+00:00", "day": "2026-10-19"}
{"url": "http://127.0.0.1:8765/private/inbox", "sent": false, "reason": "robots.txt disallow", "ts": "2026-10-19T12:09:26+00:00", "day": "2026-10-19"}
{"url": "http://127.0.0.1:8765/profile/5", "sent": false, "reason": "daily cap reached", "ts": "2026-10-19T12:09:32+00:00", "day": "2026-10-19"}
To wire it into an agent, register agent.get as the only HTTP tool the model can call, and treat Blocked as a normal tool result ("not allowed, stop") rather than an error to retry around.
The honest caveats
This is the part I care about most.
- Politeness is not permission. If a platform's terms forbid automated access, a slow and well-labelled bot still breaks them. This toolkit reduces collateral damage. It doesn't make prohibited automation acceptable.
- Prefer the official API. Meta, X, LinkedIn and most large platforms publish APIs with explicit quotas. Use them. Rate limits you are told about beat rate limits you guess.
- Logged-in scraping is a different risk. robots.txt governs crawlers on public pages. Once your agent uses a real person's session, the platform's account rules apply, and those usually prohibit automated access regardless of pace. The account you lose may belong to your client.
- Human pace is not a disguise. Jitter exists to avoid hammering servers, not to evade detection. The honest User-Agent is deliberate. If you're tempted to remove it, that's a sign the task shouldn't be automated.
- This covers HTTP, not browsers. If your agent drives a headless browser, every page load fires dozens of sub-requests. Put the bucket in front of navigations and clicks, and count each as an action.
- It's single-process. Run five workers and you get five budgets. For a fleet, move the bucket and counter into Redis or your database.
Checklist before your agent touches someone else's site
- [ ] Is there an official API? Use it instead.
- [ ] Do the terms of service allow automated access for this use?
- [ ] Is every outbound request going through one rate-limited client?
- [ ] Does the User-Agent name the bot and give a working contact?
- [ ] Is robots.txt checked, including Crawl-delay?
- [ ] Does it honour Retry-After and back off on 429 and 503?
- [ ] Is there a hard daily cap that survives restarts?
- [ ] Can you reconstruct every action from the log?
- [ ] If it runs inside a user's logged-in session, has the user agreed and do the platform rules allow it?
Rate limiting is only half of making an agent safe to deploy. The other half is controlling what the model is allowed to decide in the first place. At Axionry we build custom AI agents for teams that need this kind of control, and the same thinking runs through our LLM security work: prompt injection defence, tenant isolation, validated output and least-privilege tool access. A polite HTTP client is a good start. An agent that can only reach the tools it needs is better.
Sources: The Next Web, "Meta blames Claude tool for mistaken No Kings Instagram ban" via AI Weekly; Python docs for urllib.robotparser; RFC 9110 on Retry-After.
Top comments (0)