Writing a scraping client that respects API rate limits

A rate limit is part of an API contract. Treating 429 as an exception to hide or as a signal to retry immediately turns a temporary throttle into a longer outage. A client should know which endpoint it is calling, pace requests before the limit, and use bounded backoff when the server still rejects a request.

The public MESSORA limits differ by operation. /scrape allows 10 requests per minute on the free plan. /crawl also allows 10 enqueues per minute, /search allows 2 enqueues per minute per tenant, and /account/usage allows 30 requests per minute. /batch scales its enqueue limit with the plan: 10, 60, 120, or 300 requests per minute for free, starter, growth, and scale.

Back off instead of looping

import os
import random
import time

import requests


API = "https://api.messora.dev"
HEADERS = {"X-API-Key": os.environ["MESSORA_API_KEY"]}
RETRYABLE = {429, 500, 502, 503, 504}


def post_with_backoff(
    url: str,
    *,
    headers: dict,
    payload: dict,
    attempts: int = 5,
) -> requests.Response:
    delay = 1.0
    for attempt in range(attempts):
        response = requests.post(
            url,
            headers=headers,
            json=payload,
            timeout=30,
        )
        if response.status_code not in RETRYABLE:
            response.raise_for_status()
            return response

        if attempt == attempts - 1:
            response.raise_for_status()

        # Jitter prevents several workers from retrying on the same tick.
        time.sleep(delay + random.uniform(0, 0.25))
        delay = min(delay * 2, 30.0)

    raise AssertionError("unreachable")

The retry set in this example is intentionally narrow. A 401 or 403 is a credential problem, and a 422 is invalid input; repeating either request does not repair it. A 429 means the client should slow down. A 5xx may be transient, but the caller still needs a maximum attempt count and a visible final error.

Pace before the server answers

Backoff only helps after a rejection. For a known limit, pace enqueues in the client as well. A simple worker can keep one timestamp per endpoint and sleep until the next permitted slot. A shared scheduler is more accurate when several processes use the same API key; independent workers otherwise each believe they are below the limit while their combined traffic exceeds it.

Do not apply the /search limit to /scrape or treat all operations as interchangeable. Search starts a premium multi-source job and is limited to two enqueues per minute. If an agent asks three search questions in a short loop, queue the third instead of launching a burst and depending on retries to serialize it.

Poll jobs at the documented cadence

/batch, /crawl, and /search return a job_id and run asynchronously. The recommended polling interval for GET /jobs/{job_id} is 2 seconds. Keep the polling loop separate from enqueue throttling: the limit on creating work and the protection against excessive status reads solve different problems.

TERMINAL = {"SUCCESS", "FAILURE", "REVOKED"}


def wait_for_job(job_id: str, headers: dict, timeout_s: int = 600) -> dict:
    deadline = time.monotonic() + timeout_s
    while time.monotonic() < deadline:
        response = requests.get(
            f"https://api.messora.dev/jobs/{job_id}",
            headers=headers,
            timeout=30,
        )
        if response.status_code == 429:
            time.sleep(5)
            continue
        response.raise_for_status()
        job = response.json()
        if job["status"] in TERMINAL:
            return job
        time.sleep(2)

    raise TimeoutError(f"job {job_id} exceeded {timeout_s}s")

A 429 during polling should not be interpreted as a failed scrape. Preserve the job_id, slow the next read, and continue until your overall deadline. If the client process restarts, persist the identifier so it can resume polling instead of submitting a duplicate job.

Make throttling observable

Count attempts, 429 responses, time spent sleeping, and jobs that hit the overall deadline. Include the endpoint and plan in metrics, never the API key. A rising 429 count on /batch may mean the plan limit is too low for the workload; the same count on /search usually means the caller needs a queue or a cache.

The goal is not to make retries invisible. It is to make them bounded, attributable, and rare. A client that paces work, honors the operation-specific limits, and polls at a controlled interval spends its request budget on extraction instead of fighting the service that performs it.