Scraping 50 URLs in one job instead of 50 sequential requests

#python #api #webscraping #performance

A loop over 50 URLs calling a synchronous scrape endpoint has a wall-clock cost equal to the sum of every page's latency. With an average of 4 seconds per page, that is 200 seconds of your process sitting in requests.post, plus a rate limit you will hit around request ten.

POST /batch moves the fan-out server-side. You submit a list, get a job identifier back, and poll once.

Submit the list

import os
import requests

API = "https://api.messora.dev"
HEADERS = {"X-API-Key": os.environ["MESSORA_API_KEY"]}

urls = [
    "https://example.com/blog/post-1",
    "https://example.com/blog/post-2",
    # ... up to 50
]

resp = requests.post(
    f"{API}/batch",
    headers=HEADERS,
    json={"urls": urls, "formats": ["markdown"]},
    timeout=30,
)
resp.raise_for_status()             # 202
job_id = resp.json()["job_id"]

The cap is 50 URLs per request. A list of 51 returns 422, not a truncated job — the validation is strict so you never silently lose the tail of your input.

Chunking a larger workload

from itertools import islice

BATCH_MAX = 50


def chunked(iterable, size):
    it = iter(iterable)
    while chunk := list(islice(it, size)):
        yield chunk


def submit_all(all_urls: list[str]) -> list[str]:
    job_ids = []
    for chunk in chunked(all_urls, BATCH_MAX):
        resp = requests.post(
            f"{API}/batch",
            headers=HEADERS,
            json={"urls": chunk, "formats": ["markdown"]},
            timeout=30,
        )
        resp.raise_for_status()
        job_ids.append(resp.json()["job_id"])
    return job_ids

Deduplicate before chunking. Every duplicate URL is a charged page, and a list built by concatenating sitemap fragments routinely contains the same canonical URL three times under different query strings.

Rate limits scale with plan

/batch is limited per account by plan, unlike /scrape which is a flat 10 per minute:

Plan	Requests per minute
free	10
starter	60
growth	120
scale	300

The limit counts enqueues, not URLs. On the free plan, 10 batches per minute is 500 URLs per minute of submitted work — the constraint is almost never the enqueue rate, it is the credit balance.

Collecting results

job = wait_for_job(job_id)          # same polling loop as /crawl

ok, failed = [], []
for item in job["results"]:
    (ok if item["scrape_status"] == "success" else failed).append(item)

print(f"{len(ok)} succeeded, {len(failed)} failed, {job['credits_used']} credits")

for item in failed:
    print(item["url"], item["scrape_status"], item.get("error"))

results preserves one entry per submitted URL, each with its own scrape_status. Batch jobs report SUCCESS when the job ran, not when every page worked — a batch where 40 of 50 pages were blocked is a successful job with 10 credits charged.

Partial failure is the normal case

At 50 URLs from a mixed set of domains, expect a handful of blocked_antibot and timeout entries. Design around it:

def retry_failures(job: dict) -> list[str]:
    """URLs worth a second attempt. Extraction failures are not."""
    retriable = {"blocked_antibot", "timeout"}
    return [
        item["url"]
        for item in job["results"]
        if item["scrape_status"] in retriable
    ]

extraction_failed is not retriable — the fetch already worked, so a second identical request produces the same empty conversion. Those URLs need formats: ["raw"] and a manual parse, or they need dropping.

Choosing between batch and crawl

Both return a job_id and both poll through GET /jobs/{job_id}, but they answer different questions:

/batch — you already know the URLs. Sitemap entries, a database column, a list of competitor product pages.
/crawl — you know one URL and want the reachable set from it, filtered by url_regex.

Running a crawl when you have an explicit list wastes discovery work and gives you less control over which pages are visited. Running a batch when you do not have the list means writing your own link discovery, which is the part /crawl already does with a page cap and a frontier filter.

DEV Community