A Dataset containing text chunks can look like a complete website export even when it contains only a handful of pages. That ambiguity matters for search, migrations, and retrieval: a missing answer may reflect a crawl limit rather than missing source content.
Sitemap to Knowledge build 0.3.1 collects bounded public pages from buyer-authorized websites and creates source-linked text chunks. It does not generate embeddings or judge semantic correctness. Its useful contract combines the text with source identity, content hashes, explicit crawl gaps, and run-level delivery reconciliation.
Begin with the website scope
The installed public input accepts one to ten website values:
{
"items": ["docs.apify.com", "example.com"],
"maxConcurrency": 2
}
This example is illustrative and does not assert current collection results. Values normalize to origins and equivalent origins are deduplicated. Use items for new inputs. The legacy websites alias is used only when items is absent; sending both creates an unnecessary ambiguity for someone maintaining the integration later.
Concurrency controls parallel sites and accepts one to twenty. Each site remains independently bounded. Raising concurrency does not remove the caps or turn the operation into an exhaustive crawler.
Understand the discovery and chunk boundaries
The collector tries conventional sitemap locations. It attempts sitemap_index.xml after sitemap.xml only when the first location is definitively missing. A homepage fallback is reserved for the case where both conventional locations are definitively missing. A timeout or a forbidden response is not equivalent to evidence that no sitemap exists.
Within a site, the installed limits are ten child sitemaps, twenty-five pages, and two hundred chunks. Discovery stays within the supported same-site scope and one generation of child sitemaps. These boundaries should be visible in a corpus manifest, especially for a documentation site with many sections.
Text is split at word boundaries into approximately one-thousand-character chunks. This is deterministic size-based chunking, not semantic segmentation. It does not establish that each chunk contains a complete argument, that adjacent sections overlap, or that an embedding model will retrieve the correct context.
Preserve coverage as data
Each delivered chunk includes source identity and a content hash, while partial and partialReasons communicate collection limits. crawlStats retains counts related to discovery, fetched pages, skipped content, failures, truncation, and caps. Keep these fields in the index manifest instead of exporting only the text column.
The content hash is useful for identifying repeated content. It does not prove that the entire site was collected, that the source is accurate, or that the text is licensed for reuse. An observation timestamp also describes this collection, not the last time the author checked every statement on the page.
A partial chunk is not automatically a failed or free outcome. Delivered chunks marked partial remain billable, including the documented homepage fallback after definitive sitemap absence. Free invalid-input or collection-failure outcomes have different delivery semantics. An ambiguous delivery needs reconciliation; it must not be advertised as free because a local process stopped waiting.
Accept a corpus with a manifest
For a practical acceptance exercise, use a small site with a known sitemap, a larger fixture that reaches a page cap, and a controlled site with both conventional sitemap paths returning 404. Inspect the resulting coverage states and confirm that fallback content stays explicitly labeled. Add two origin-equivalent inputs to check that deduplication does not hide an attempted scope difference.
These are proposed checks, not newly executed observations. Before embedding anything, decide whether the delivered scope is sufficient for the intended question. If the scope is partial, retain that qualification in any downstream answer or migration checklist. A bounded corpus can still be useful; its usefulness depends on knowing which evidence it contains and which questions the missing pages prevent it from answering.
Cost and source boundary
The live FREE-tier event prices checked on 20 October 2026 are $0.001 per result-found event and $0.005 per automatic Actor-start event. The start event is memory-scaled: one event per GB, with a minimum of one event. A free result outcome therefore does not make an entire run free. Other account tiers have their own rates; check the live pricing panel before execution. These prices describe the installed contract, not measured customer revenue or a new runtime test.
Inspect the Actor, input, and current pricing.
AI-assisted drafting; technical statements reviewed against the installed Actor source and current API metadata. Examples are illustrative unless explicitly identified as saved observations.
Top comments (0)