Polling an asynchronous crawl job without hammering the API

#python #api #webscraping #async

A breadth-first crawl over 50 pages does not fit inside an HTTP request. The MESSORA /crawl endpoint returns 202 Accepted with a job identifier and runs the work on a queue, which means your client needs a polling loop that terminates.

Most polling loops written in a hurry share the same three bugs: they treat an unknown state as terminal, they poll at a fixed interval that trips rate limits, and they never time out. Here is one that handles all three.

Enqueue

import os
import time
import requests

API = "https://api.messora.dev"
HEADERS = {"X-API-Key": os.environ["MESSORA_API_KEY"]}


def start_crawl(seed: str, max_pages: int = 25) -> str:
    resp = requests.post(
        f"{API}/crawl",
        headers=HEADERS,
        json={
            "url": seed,
            "max_pages": max_pages,
            "only_main_content": True,
            "url_regex": r"/docs/.*",
        },
        timeout=30,
    )
    resp.raise_for_status()          # 202 on success
    return resp.json()["job_id"]

url and max_pages are both required; there is no implicit page budget. max_pages caps at 50. url_regex filters discovered links before they enter the frontier, which is the difference between crawling a documentation tree and crawling an entire marketing site.

The four job states

GET /jobs/{job_id} reports Celery task states, in uppercase:

State	Terminal	Meaning
`PENDING`	no	Queued, no worker has picked it up
`STARTED`	no	Worker is crawling
`SUCCESS`	yes	`results` is populated
`FAILURE`	yes	`error` carries a sanitized message
`REVOKED`	yes	Cancelled; no partial results exposed

PENDING is ambiguous in Celery generally — it also covers task IDs that never existed. Here it does not, because the job row is looked up per tenant before the queue is consulted. A 404 means the job belongs to a different account or was never created; it never means "still starting".

A polling loop that terminates

TERMINAL = {"SUCCESS", "FAILURE", "REVOKED"}


def wait_for_job(job_id: str, timeout_s: int = 600) -> dict:
    deadline = time.monotonic() + timeout_s
    delay = 2.0

    while time.monotonic() < deadline:
        resp = requests.get(f"{API}/jobs/{job_id}", headers=HEADERS, timeout=30)

        if resp.status_code == 429:
            time.sleep(delay)
            delay = min(delay * 2, 30.0)
            continue

        resp.raise_for_status()
        job = resp.json()

        if job["status"] in TERMINAL:
            return job

        time.sleep(delay)
        delay = min(delay * 1.5, 15.0)

    raise TimeoutError(f"job {job_id} still {job['status']} after {timeout_s}s")

Three decisions worth naming:

Exponential backoff with a ceiling. A crawl of 50 pages takes minutes, not seconds. Polling every 500 ms produces hundreds of pointless requests and eventually a 429.
429 does not consume the retry budget. It is a signal to slow down, not a job failure.
time.monotonic(), not time.time(). A clock adjustment mid-crawl should not extend or truncate your timeout.

Reading the results

job = wait_for_job(start_crawl("https://example.com/docs/"))

if job["status"] == "FAILURE":
    raise RuntimeError(job["error"])

print(f"pages crawled: {job['pages_crawled']}")
print(f"stopped because: {job['stopped_reason']}")
print(f"credits: {job['credits_used']}")

for item in job["results"]:
    if item["scrape_status"] == "success":
        print(item["url"], len(item["markdown"]), "chars")
    else:
        print(item["url"], "skipped:", item.get("reason"))

stopped_reason is the field that answers "why did I get 12 pages instead of 25". Canonical values are completed, timeout, token_cap, and discovery_failed. A crawl that returns fewer pages than requested with stopped_reason: "completed" simply ran out of links matching your regex — the crawl worked, the frontier was empty.

Per-item statuses matter

results is a list where each entry carries its own scrape_status. A crawl is rarely all-or-nothing: a documentation tree with two PDF downloads and one page behind bot protection returns SUCCESS at the job level with three non-success items inside.

Credits are charged per successfully crawled page, so a job that discovers 25 URLs and extracts 22 costs 22 credits. Counting len(results) overstates the spend and understates your failure rate.

Rate limits in practice

/crawl allows 10 enqueues per minute per account. /jobs/{job_id} is bounded separately as anti-polling protection. If you are orchestrating dozens of crawls, queue the enqueues client-side rather than firing them in a loop and catching 429 — the backoff you write is always cheaper than the one the server imposes.

DEV Community