Changing crawl providers is a workflow migration, not a base-URL edit.
Firecrawl's current Crawl API starts a job, returns a job ID, and exposes status and paginated results. Its Python SDK can wait for a crawl and return all results. Nstdata's docs describe async crawl submission, status polling, paginated page results, and storage references for large Markdown artifacts. The route examples differ across sections of the Nstdata docs, so verify the exact route and response schema in your account documentation before deploying.
This post focuses on a safer seam: normalize both providers into one application-owned document type before chunking or indexing.
The code is illustrative and has not been run against either paid API. Verify current API contracts before production use.
1. Define an internal document contract
Keep provider response fields out of the rest of the RAG pipeline:
python
from dataclasses import dataclass
@dataclass(frozen=True)
class Document:
url: str
canonical_url: str | None
title: str | None
markdown: str
status_code: int | None
fetched_at: str
provider: str
Include enough provenance to inspect bad retrieval later. In production, the job ID, original input URL, artifact reference, response metadata, and retry count may belong in a separate ingestion record.
2. Normalize at the adapter boundary
The following function consumes a page metadata dictionary and the already-resolved Markdown text:
`def normalize_page(page: dict, markdown: str, provider: str) -> Document:
url = page.get("url") or page.get("sourceURL") or page.get("sourceUrl")
if not isinstance(url, str) or not url.startswith(("http://", "https://")):
raise ValueError("Page has no absolute source URL")
body = markdown.strip()
if not body:
raise ValueError(f"Empty page content: {url}")
metadata = page.get("metadata") or {}
status = metadata.get("statusCode", page.get("statusCode"))
return Document(
url=url,
canonical_url=metadata.get("canonicalUrl") or page.get("canonicalUrl"),
title=metadata.get("title") or page.get("title"),
markdown=body,
status_code=int(status) if status is not None else None,
fetched_at=page.get("fetchedAt") or page.get("fetched_at") or "",
provider=provider,
)
`
The mapper is intentionally small. Extend it only for fields your downstream logic uses. Preserve the raw provider response in an audit store so a schema change or parser issue is diagnosable.
For Nstdata, resolve markdownRef before calling the normalizer. The reference token identifies stored output; it is not the Markdown itself. For Firecrawl, normalize the SDK document or each direct API result page. The Firecrawl SDK manages pagination when using its crawl-and-wait method; callers of the direct API should follow next until it is absent. Firecrawl response handling · Nstdata artifact reads
3. Persist job state so the migration can resume
Use a durable record per crawl run:
text
run_id
provider
input_manifest_hash
job_id
submitted_at
last_status
last_cursor
accepted_count
rejected_count
terminal_error
A migration runner can then follow this sequence:
- Load a fixed, versioned URL manifest.
- Submit a bounded crawl with an explicit page cap and path filters.
- Persist the job ID before polling.
- Poll with bounded backoff until a terminal state.
- Fetch result pages until pagination ends.
- Resolve large artifact references and normalize each page.
- Record rejected URLs with a reason instead of silently dropping them.
- Write accepted documents into a staging index.
- Compare the URL set and retrieval quality with the existing index.
- Switch a small cohort, monitor it, then expand.
A timeout between submission and polling should resume from the saved job ID. Restarting the entire crawl can duplicate billable work. Nstdata currently documents no external webhooks, so async completion needs a poller or a separate event mechanism.
4. Treat acceptance as a separate status
HTTP status and crawl task status do not decide whether a page belongs in your RAG index. A 404 can be retrieved successfully but be invalid documentation; a 200 response can contain an access challenge, empty shell, or partial JavaScript content.
Define an application-level validation result:
python
def accept_document(doc: Document) -> tuple[bool, str]:
if doc.status_code in (403, 404):
return False, f"target_http_{doc.status_code}"
if len(doc.markdown.strip()) < 80:
return False, "content_too_short"
if "access denied" in doc.markdown[:500].lower():
return False, "challenge_or_block_page"
return True, "accepted"
The threshold and block-page checks are examples to customize; a generic length rule will not fit every site. Add source-specific checks for required headings, code sections, or product fields. Keep terminal failures separate from retryable failures.
5. Run the same corpus through both providers
Use 100 representative URLs: static docs, JavaScript pages, long code samples, tables, redirects, and pages that previously failed. Compare:
- accepted page count and missing URL set;
- canonical URLs and duplicate handling;
- heading, code, table, and link retention;
- p50/p95 time to validated content;
- retry count, artifact reads, and total billed cost;
- staging-index answer support and citation URLs.
Do not compare byte-for-byte Markdown. The acceptance criterion is whether the same useful facts survive and downstream questions still cite the right source.
6. Roll out with a rollback path
Keep the existing adapter behind a feature flag. Start with a small scheduled cohort, compare one complete crawl cycle, then test page deletion and update behavior before moving all traffic. Include stale-vector deletion in the cutover plan: switching the fetcher without removing obsolete chunks can make the new system look wrong even if the new page extraction is correct.
A short proof of concept demonstrates a happy path. Production parity also includes filters, polling, artifact storage, retries, deletion, alerting and rollback. Calculate engineering cost from the pilot instead of treating the API-call replacement as the whole migration.
Top comments (0)