Crawl an Entire Website With One API Request: A Production Guide

TL;DR

• Submit a bounded site crawl with POST /api/v1/crawl, then poll /api/v1/crawl/status/{crawlId} and paginate /api/v1/crawl/pages/{crawlId}.
• Always set maxDepth, maxPages, include/exclude patterns, query handling, output formats, and timeouts explicitly.
• Treat a completed task as a transport state, not proof of dataset completeness.
• Persist crawl IDs, pagination cursors, page identity, failure reasons, and artifact references separately.
• Start with a small pilot and measure unique accepted pages before raising limits.

“Crawl an entire website with one API request” is accurate only if “one request” means job submission.

A production crawl has a longer lifecycle:

submit
  → persist crawl ID
  → poll status
  → paginate pages
  → retrieve large artifacts
  → canonicalize and deduplicate
  → validate coverage

This tutorial implements that lifecycle around the current Nstproxy Crawl REST surface. The requests require your own API key and an authorized public target.

1. Define the Crawl Boundary

Before writing code, define which URLs belong in the dataset.

For a documentation crawl, a reasonable contract might be:

seed: https://example.com/docs/
allowed_host: example.com
include_paths:
  - /docs/
exclude_paths:
  - /docs/archive/
  - /docs/search/
ignore_query: true
max_depth: 3
max_pages: 100
formats:
  - markdown

maxDepth limits link hops from the seed. maxPages limits total work. Use both: a shallow graph can still be extremely wide.

Only enable ignoreQuery when query parameters do not identify meaningful variants, locales, versions, or pagination states.

2. Submit One Site-Crawl Request

Nstproxy Crawl documents POST /api/v1/crawl for site-wide submission.

curl --request POST \
  --url 'https://api.nstproxy.com/api/v1/crawl' \
  --header "x-api-key: $NSTPROXY_API_KEY" \
  --header 'Content-Type: application/json' \
  --data '{
    "url": "https://example.com/docs/",
    "formats": ["markdown"],
    "maxDepth": 3,
    "maxPages": 100,
    "includeUrls": ["*example.com/docs/*"],
    "excludeUrls": [
      "*example.com/docs/archive/*",
      "*example.com/docs/search/*"
    ],
    "ignoreQuery": true,
    "onlyMainContent": true,
    "timeout": 60000
  }'

The expected response contains an ID and a processing state. Store the exact returned ID. Never manufacture, trim, or parse meaning from it.

Request only the formats the downstream application needs. Markdown is useful for RAG and text analysis; HTML is better when selectors or DOM semantics are required. Screenshots and PDF are valuable for visual or archival requirements but increase artifact handling.

3. Poll With a Deadline and Jitter

The current status route is:

GET /api/v1/crawl/status/{crawlId}

A dependency-free Python poller can use the standard library:

from __future__ import annotations

import json
import os
import random
import time
import urllib.error
import urllib.request


BASE_URL = "https://api.nstproxy.com"
API_KEY = os.environ["NSTPROXY_API_KEY"]
TERMINAL_STATES = {"completed", "failed", "cancelled"}


def get_json(path: str) -> dict:
    request = urllib.request.Request(
        f"{BASE_URL}{path}",
        headers={"x-api-key": API_KEY},
    )
    with urllib.request.urlopen(request, timeout=30) as response:
        return json.load(response)


def poll_crawl(crawl_id: str, deadline_seconds: int = 900) -> dict:
    deadline = time.monotonic() + deadline_seconds
    attempt = 0

    while time.monotonic() < deadline:
        payload = get_json(f"/api/v1/crawl/status/{crawl_id}")
        task = payload.get("data", {}).get("data", payload.get("data", {}))
        status = task.get("status")

        print(
            "status=", status,
            "completed=", task.get("completed"),
            "pending=", task.get("pending"),
            "failed=", task.get("failed"),
        )

        if status in TERMINAL_STATES:
            return task

        delay = min(30.0, 2 ** min(attempt, 5))
        time.sleep(delay + random.uniform(0, delay * 0.2))
        attempt += 1

    raise TimeoutError(f"crawl {crawl_id} exceeded client deadline")

Do not poll continuously. Use bounded exponential backoff, jitter, and an overall client deadline. If the API supplies Retry-After, honor it.

Inspect the response body rather than relying only on HTTP 200. Task-level failure information can exist inside a successfully delivered response envelope.

4. Paginate Every Crawled Page

The current page-results route is:

GET /api/v1/crawl/pages/{crawlId}

The response may include a nextCursor. Continue until the cursor is absent.

import urllib.parse


def iter_crawled_pages(crawl_id: str, page_size: int = 50):
    cursor = None

    while True:
        query = {"limit": str(page_size)}
        if cursor:
            query["cursor"] = cursor

        path = (
            f"/api/v1/crawl/pages/{crawl_id}?"
            f"{urllib.parse.urlencode(query)}"
        )
        payload = get_json(path)
        result = payload.get("data", {})

        for page in result.get("data", []):
            yield page

        cursor = result.get("nextCursor")
        if not cursor:
            break

Stopping after the first result response is a common reason a completed crawl appears incomplete.

Persist pagination cursors as checkpoint state, not as document identity. A page should be identified using its normalized final or canonical URL plus a content hash when appropriate.

5. Resolve Large Artifact References

Large outputs may be represented by fields such as markdownRef, htmlRef, rawDataRef, screenshotRef, or pdfRef.

The documented storage route is:

GET /api/v1/crawl/storage/read?st={ref}

Treat the reference as opaque. Do not decode, reconstruct, normalize, or edit it.

Only resolve artifacts needed by the downstream job. A discovery pipeline may require URLs and metadata first, while a later worker retrieves Markdown for accepted pages.

6. Canonicalize Without Destroying Meaning

URL normalization is necessary, but aggressive normalization can merge distinct documents.

Safe operations often include lowercasing the hostname, removing the fragment, normalizing default ports, and resolving dot segments. Query parameters require domain knowledge. Tracking parameters may be disposable; locale, variant, version, and pagination parameters may be essential.

Use a standards-aware parser. The WHATWG URL Standard documents modern parsing behavior.

A simple canonical key might look like:

from urllib.parse import parse_qsl, urlencode, urlsplit, urlunsplit


DROP_QUERY_KEYS = {"utm_source", "utm_medium", "utm_campaign", "ref"}


def canonical_key(url: str) -> str:
    parts = urlsplit(url)
    query = [
        (key, value)
        for key, value in parse_qsl(parts.query, keep_blank_values=True)
        if key.casefold() not in DROP_QUERY_KEYS
    ]

    return urlunsplit((
        parts.scheme.casefold(),
        parts.netloc.casefold(),
        parts.path or "/",
        urlencode(sorted(query)),
        "",
    ))

This example is intentionally conservative. Extend the drop list only after validating the target’s URL semantics.

7. Validate Coverage, Not Just Completion

A completed status means the job stopped running. It does not prove that the desired corpus was captured.

Compare the accepted result set with a sitemap, navigation inventory, CMS export, or manually labelled sample. The Sitemap protocol defines the common XML format, while the Robots Exclusion Protocol defines robots.txt behavior.

Track:

expected URLs
discovered URLs
processed pages
meaningful-content pages
accepted unique pages
duplicates
redirects
failures by reason
budget consumption by path family

If the crawl reaches maxPages, report the dataset as budget-limited rather than complete. If one path family consumes most of the budget, tighten includeUrls and excludeUrls before raising the limit.

8. Handle Failures by Layer

Do not retry the whole site because several pages failed.

Separate:

• submission errors, such as invalid request fields;
• task errors, such as a terminal job failure;
• target errors, such as timeout or access denial;
• content failures, such as empty text or a soft 404;
• quality failures, such as duplicate or wrong-locale content.

Retry transient target failures individually with a strict ceiling. Authentication errors, disallowed targets, parser failures, and empty templates require different remediation.

For recurring crawls, keep canonical URLs and content hashes. Reprocess changed pages, add new pages, and remove deleted pages from downstream indexes.

9. Protect the Crawl Boundary

Accept only authorized public HTTP or HTTPS targets. If users provide seeds, block localhost, private and link-local networks, cloud metadata endpoints, unsafe ports, and redirects outside the allowed host.

The OWASP SSRF guidance provides a useful threat model. Also respect terms, privacy obligations, copyright, robots rules, and target capacity.

Nstproxy’s scraping-versus-crawling guide explains the difference between link discovery and page extraction. The Nstproxy Crawl pricing page describes current billing options; evaluate them using cost per accepted unique page.

10. Resume Jobs Without Duplicating Work

A crawler client should be restartable. Store the submission configuration and crawl ID before polling begins, then checkpoint the last successfully consumed pagination cursor after each result page is committed.

The order of operations matters:

fetch result page
  → validate page envelopes
  → commit accepted records
  → commit rejection records
  → save next cursor

Saving the cursor before committing records can lose data after a crash. Committing records before saving the cursor can replay a page, so downstream writes should be idempotent.

Build an idempotency key from stable document attributes such as the canonical URL, content hash, and crawl configuration version. A repeated result can then update the same logical version instead of creating a duplicate.

Do not automatically submit a new crawl when a poller restarts. First load the stored crawl ID and query its status. Resubmission can create two active jobs that discover and process the same site independently.

For recurring crawls, distinguish a run ID from a page ID. The run records when and how discovery occurred; the page identity connects versions of the same canonical document across runs.

11. Control Concurrency at Two Layers

Site-crawl systems have two different forms of concurrency: the number of crawl jobs submitted and the number of page operations performed inside each job. A client-side queue controls the first; the provider and selected plan usually constrain the second.

Submitting many site jobs simultaneously can concentrate traffic on the same host even when every individual job has a conservative page limit. Group work by registrable domain, enforce a per-domain job ceiling, and add jitter between recurring runs.

Monitor queue wait time separately from page-processing latency. A slow job may be waiting for capacity rather than struggling with the target page. This distinction prevents unnecessary retries and helps determine whether the application needs a higher service limit or a better scheduling policy.

Rate limits should be handled as flow control, not as generic failures. Honor Retry-After when available, reduce submission pressure, and preserve the existing crawl ID. Repeatedly creating new jobs after a limit response makes recovery less predictable.

12. Test the Crawl Configuration Before Production

Keep a small regression set containing URLs that should be included, URLs that should be excluded, equivalent tracking variants, meaningful query variants, redirects, soft 404s, and JavaScript-dependent pages. Run the configuration against this set whenever URL rules change.

The test should assert both positive and negative behavior. Confirm that required documentation pages remain eligible, but also prove that calendars, account routes, internal search pages, and unwanted file types stay outside the crawl. Negative tests are particularly valuable because an overly broad pattern may not fail visibly; it simply consumes the page budget on low-value URLs.

Record the crawl configuration as a versioned object with every run. When coverage changes, the team can distinguish a website change from an intentional configuration update. This also makes historical datasets explainable and lets a failed rollout return to a known boundary without guessing which combination of patterns was previously used.

Final Take

One API request can start an entire-site workflow, but a reliable dataset still requires explicit boundaries and verification.

Set depth and page limits. Constrain paths. Normalize URLs carefully. Poll with a deadline. Retrieve every result page. Resolve only required artifacts. Then compare accepted unique pages with a trusted inventory.

The pilot crawl is where most costly mistakes should be found. Keep it small until the URL distribution, result quality, and failure policy are correct.

FAQ

Can I crawl an entire website with one REST request?

You can submit a bounded site-wide job with one REST request. The asynchronous workflow still requires status polling and paginated result retrieval afterward.

Why do I need both maxDepth and maxPages?

maxDepth limits how far the crawler follows link chains, while maxPages caps total work when a level contains many URLs. They protect against different graph shapes.

Should query parameters always be ignored?

No. Ignore query parameters only when they do not change meaningful content. Locale, product variant, version, search, and pagination parameters may identify distinct pages.

How should I store crawl progress?

Persist the crawl ID, request configuration, current task state, last pagination cursor, result counts, failure counts, and timestamps. Keep transport checkpoints separate from page identity.

How do I know whether the crawl is complete?

Compare accepted canonical pages with a sitemap or known inventory, inspect failures and path coverage, and confirm that every paginated result page has been retrieved.