You can get Markdown from a URL with a short Crawl4AI script. The production decision starts when that script becomes a recurring job: who runs Chromium, limits concurrency, manages proxies, persists task state, and fixes failed pages?
This guide compares self-hosted Crawl4AI with Nstdata Crawl, then separates both from Crawl4AI Cloud. No credentials or controlled test run were available for this article, so the code follows the official docs and performance claims are left to a reproducible test on your URLs.
First, separate the deployment options
Crawl4AI is an open-source Python crawler with an in-process library and a self-hosted Docker server. Its current documentation also describes Crawl4AI Cloud, a hosted API. This makes three options:
| Option | You run | You configure |
|---|---|---|
| Crawl4AI library | Python process and browser | Browser, crawl logic, proxies, queue and monitoring |
| Crawl4AI Docker | Container and browser runtime | Server authentication, resources, upgrades and job recovery |
| Crawl4AI Cloud | Your client application | API requests and Cloud plan/limits |
| Nstdata Crawl | Your client application | API credentials, scope, task polling and downstream validation |
The rest of this post focuses on the library/Docker versus Nstdata API comparison. Check Crawl4AI Cloud's current plans and endpoints separately if you want a hosted version of the Crawl4AI workflow.
Minimal Python example
Both snippets request Markdown for https://example.com/. They are documentation-based and not executed. Install the packages and set NSTDATA_API_TOKEN for Nstdata before trying these in your environment.
Crawl4AI in your Python process
pip install crawl4ai
crawl4ai-setup
`import asyncio
from crawl4ai import AsyncWebCrawler
async def main():
async with AsyncWebCrawler() as crawler:
result = await crawler.arun(url="https://example.com/")
if not result.success:
raise RuntimeError(result.error_message)
print(result.markdown)
asyncio.run(main())`
This is useful when you need Python-level control. BrowserConfig controls browser-wide behavior; CrawlerRunConfig carries settings for a run. Crawl4AI documents CSS/XPath extraction and LLM-based extraction as options on top of Markdown conversion.
Nstdata Crawl through its Python SDK
pip install nstdata-ai-crawl
`import os
from nstdata_ai_crawl import NstDataClient, ScrapeRequestDto, Format
with NstDataClient(os.environ["NSTDATA_API_TOKEN"]) as client:
result = client.submit_scrape_task_sync(ScrapeRequestDto(
url="https://example.com/",
formats=[Format.MARKDOWN],
timeout=60000,
onlyMainContent=True,
))
markdown = result.data.get_markdown()
if not markdown:
raise RuntimeError("Crawl returned no Markdown")
print(markdown)`
The SDK method and response helper above come from Nstdata's current docs. Confirm the SDK version and endpoint contract before deploying; its public documentation shows multiple route examples. Nstdata Crawl SDK docs.
The core code is short for both. The real migration delta lives in retries, timeouts, persistent job state, result shape and operational ownership.
What breaks between prototype and production?
1. Browser resources and concurrency
Every browser page consumes CPU, memory and network. Raising concurrency can increase throughput until the target rate limit or your worker memory becomes the bottleneck. Crawl4AI's multi-URL dispatcher docs describe memory-adaptive dispatch, concurrency limits, rate limiting and monitoring helpers. With self-hosting, you still size the machine pool and decide what happens when a worker dies.
Nstdata runs the browser infrastructure for API requests. Your client still needs bounded submission, account-limit awareness and a durable record of task IDs. A managed browser does not automatically give your pipeline idempotency or exactly-once ingestion.
2. Proxies and blocked pages
Crawl4AI's self-hosted configuration lets you provide proxy settings and fallback behavior. That means you select, pay for and diagnose proxy endpoints. Nstdata documents proxy coordination and region selection, with proxy traffic billed separately.
Treat a 403, 429, timeout and parser error as different states:
| Signal | Check | Response to test |
|---|---|---|
| 403 | Is this a policy block, challenge page or access restriction? | Stop if collection is not permitted; otherwise inspect allowed session/region settings |
| 429 | Did request rate exceed target limits? | Reduce concurrency and use bounded backoff |
| Timeout | Did navigation, a selector wait or the API time out? | Isolate the stage; adjust timeout or use async processing |
| 200 with empty/challenge body | Did the browser load useful content? | Validate page text and expected selectors, not status alone |
| Parse error | Did the DOM change or selector stop matching? | Save the response for diagnosis and update extraction logic |
Crawl4AI documents stealth-related browser options and proxy fallbacks; Nstdata describes fingerprint-browser handling. These are mechanisms, not guarantees of access. Test only pages you are allowed to collect.
3. Task state and recovery
For self-hosted crawling, persist your own job ID, URL, attempt count, status, result location and terminal reason. A process-local queue is easy to lose during a restart. Crawl4AI has dispatcher helpers, but you still choose the durable queue and recovery model.
Nstdata offers synchronous and asynchronous scraping. Async responses return a task ID; clients poll for status and retrieve output. Site crawls have status/progress and paginated pages. Its current docs say external webhooks are not available, so you need a polling worker for async completion. Check nested success and status; an HTTP 200 can indicate job acceptance rather than usable content.
An ingestion state machine could be as small as:
text
queued -> submitted -> processing -> validated -> indexed
\-> retryable_error -> submitted
\-> terminal_error
Make retries bounded, store the final error, and avoid retrying a permanent access denial indefinitely.
Compare Markdown quality with an acceptance test
Don't use a vague “RAG-ready” label as your test. Pick 100 representative public pages and define measurable checks:
- Expected main text appears in the output.
- Heading hierarchy is preserved.
- Code blocks, tables and links survive extraction.
- Navigation, cookie banners and unrelated content stay below an agreed noise threshold.
- Metadata includes the source URL and page title.
- Duplicate canonical pages are identified.
- Empty, blocked and partial pages are marked as failures before indexing.
Include static docs, delayed JavaScript content, a click-to-expand section, a long code sample, a table, a redirect and a permitted region-specific URL. Record p50/p95 time to a usable result, error category, retries, proxy usage, token count and operator time. For a bounded crawl, compare discovered URLs against a known inventory. A maxPages: 50 cap limits work; it does not prove the 50 desired pages were found. Nstdata documents maxDepth, maxPages, includeUrls and excludeUrls; Crawl4AI provides deep-crawl strategies.
Estimate total cost instead of comparing license fees
The Crawl4AI library itself has no software license charge. Self-hosted production costs include:
text
compute + proxies + storage/network + monitoring + engineering/operations
If average browser occupancy is 20 seconds per page, then 100,000 pages consume 2,000,000 browser-seconds, or about 556 browser-hours before parallelism, retries or idle capacity. This is an assumption for spreadsheet planning, not a benchmark. Replace it with measured runtime and your actual infrastructure rates.
Nstdata's current docs list a pay-as-you-go base rate of $1.20 per 1,000 processed requests. Proxy traffic is separate; the docs say JavaScript rendering, Markdown, PDF and screenshots are included in the base service. Some returned HTTP statuses, including 403 and 404, may still count as successfully retrieved requests for billing. System failures that retrieve no page content are not billed.
Assuming one billable request per target, no retries, no subscription discount and no proxy traffic:
| Requests/month | Nstdata base request estimate | Self-hosted Crawl4AI |
|---|---|---|
| 10,000 | $12 | Measure compute + proxy + storage + labor |
| 100,000 | $120 | Measure worker capacity + proxy + monitoring + labor |
| 1,000,000 | $1,200 | Measure scaled capacity + proxy + recovery + labor |
These are base-rate calculations, not final bills. Compare full monthly TCO divided by accepted pages for both. Add retry and proxy charges, not just successful requests. Evaluate Crawl4AI Cloud as its own hosted price model rather than mixing it with self-hosted costs.
Migration checklist
Before replacing AsyncWebCrawler calls with API calls, inventory what your current crawler actually does:
- BrowserConfig, CrawlerRunConfig, hooks, cookies and session reuse;
- CSS/XPath or LLM extraction schemas;
- URL discovery, filtering, deduplication and refresh behavior;
- proxy rules, concurrency and per-domain rate limits;
- result fields, artifact files and downstream storage;
- async job IDs, retries, alerts and rollback.
Run both systems against the same 100 URLs. Normalize outputs to an internal record such as:
json
{
"source_url": "https://example.com/docs/page",
"title": "Page title",
"markdown": "# Page title\n...",
"fetched_at": "2026-10-16T00:00:00Z",
"status": "validated"
}
This is your pipeline schema, not a vendor response. It keeps provider-specific response differences out of the vector indexing code.
Estimate migration ROI with:
text
one-time migration = implementation + dual-run validation + rollback work
monthly savings = old vendor + old maintenance - new vendor - new maintenance
payback months = one-time migration / verified monthly savings
Keep Crawl4AI for targets that depend on custom hooks or extraction behavior if the managed API doesn't cover them. A hybrid system is often simpler than rewriting one-off browser logic to force a uniform provider.
Which should you use?
Use self-hosted Crawl4AI when you need code-level control, already operate browser infrastructure, or rely on custom extraction and navigation.
Use Nstdata Crawl when the workload is mostly standard page or bounded-site collection and you want browser operations behind a managed API. Your application still owns scope, validation, polling and storage.
Consider Crawl4AI Cloud when you want Crawl4AI's hosted API. Compare current limits and prices with Nstdata on the same URL set.
The answer can vary by target. Evaluate cost per accepted page, including engineering time, on a representative pilot before migrating production traffic.
Top comments (0)