Avoiding duplicate fetches with max_age_ms

Scheduled ingestion often revisits the same pages: a product catalog every hour, documentation every morning, or a set of policy pages after a deployment. Fetching every URL from scratch wastes time when the source has not changed. The MESSORA /scrape endpoint accepts max_age_ms so a caller can opt into a recent cached result for the same request.

The default is no cache lookup. That is the safe choice for a fresh scrape. Set a positive max_age_ms only when stale content is acceptable for the job's purpose. The accepted range is 0 through 2,592,000,000 milliseconds, or 30 days.

Opt in for a bounded freshness window

import os
import requests

API = "https://api.messora.dev"
HEADERS = {"X-API-Key": os.environ["MESSORA_API_KEY"]}

response = requests.post(
    f"{API}/scrape",
    headers=HEADERS,
    json={
        "url": "https://example.com/pricing",
        "formats": ["markdown"],
        "only_main_content": True,
        "max_age_ms": 3_600_000,  # accept a result up to one hour old
    },
    timeout=90,
)
response.raise_for_status()
payload = response.json()

print(payload["markdown"])
print(payload.get("diagnostics", {}).get("from_cache", False))

A one-hour window is appropriate for a page where hourly changes are not important. A legal notice or incident page may need max_age_ms: 0, which explicitly bypasses the cache. Do not use a long freshness window just because a scrape is expensive; choose it from the data's acceptable staleness.

Know what identifies a cached result

The cache key includes the scrape request fingerprint, not only the URL. Changes to output format or extraction parameters should not be treated as the same result. Keep the request stable between runs if you want cache hits: changing only_main_content, render_js, wait_for, or the requested format can represent a different extraction.

The response exposes cache provenance in diagnostics.from_cache. Use it in metrics and in a persisted ingestion record. A cache hit returns credits_used: 0 for that request and restores the credit that was provisioned for the attempt. That lets a frequent refresh avoid paying for a duplicate fetch while still making the freshness decision explicit.

def scrape_and_record(url: str, max_age_ms: int | None) -> dict:
    payload = requests.post(
        f"{API}/scrape",
        headers=HEADERS,
        json={
            "url": url,
            "formats": ["markdown"],
            "max_age_ms": max_age_ms,
        },
        timeout=90,
    )
    payload.raise_for_status()
    data = payload.json()

    if data["scrape_status"] != "success":
        raise RuntimeError(data.get("error") or data["scrape_status"])

    return {
        "url": url,
        "markdown": data["markdown"],
        "from_cache": data.get("diagnostics", {}).get("from_cache", False),
        "credits_used": data["credits_used"],
    }

Persisting from_cache prevents an operator from mistaking a repeated cached read for a fresh observation. It also gives you a way to compare freshness policy with actual behavior when a site changes.

Do not confuse cache with a source change check

A cache hit says that the API has a completed result within the requested age. It does not prove that the origin currently serves identical bytes. If the workflow needs to detect every source update, use a fresh scrape and compare a normalized hash of the returned content. If the workflow only needs a boundedly fresh copy, the cache is cheaper and simpler.

A cache miss follows the normal scrape path. The result is stored only after a successful extraction, using the requested age as the cache TTL. Failed or incomplete fetches should remain visible as failures rather than poisoning future runs with an empty body.

Choose policy per dataset

Keep cache settings in the dataset configuration, not hidden inside a generic HTTP helper. A news feed, a product page, and a versioned API reference have different freshness requirements. Review the policy when consumers change: a value that was harmless for a search index may be wrong for an alerting system.

max_age_ms is a small parameter with a clear tradeoff. Zero favors freshness and predictable observation. A positive value favors lower latency and fewer duplicate fetches. Recording cache provenance and actual credits makes that tradeoff measurable instead of leaving it as an undocumented optimization.