Prepare Markdown for retrieval while preserving failed and partial pages

A retrieval pipeline can lose useful evidence before it generates its first embedding. Converting HTML to text is simple enough to hide operational decisions: which URL was fetched, whether a redirect changed the source, whether the response was truncated, and whether a failed fetch was accidentally stored as an empty document.

URL to Markdown build 0.1.15 produces a bounded Markdown observation from authorized public HTML. It is a collection component for a later retrieval workflow. It does not render JavaScript, call an LLM, generate embeddings, or establish that the resulting text is licensed for a particular reuse.

Make the page list explicit

This illustrative input requests two static pages and retains links:

{
  "urls": [
    "https://example.com",
    "https://www.iana.org/help/example-domains"
  ],
  "includeLinks": true,
  "maxConcurrency": 2
}

The example shows a supported request, not fresh conversion results. The input accepts up to 100 public HTTP(S) page URLs. includeLinks controls whether anchor links become Markdown links or only their text remains. Choose that setting based on the downstream document format rather than dropping links accidentally during a later cleanup step.

Normalized duplicate URLs are processed once. Keep the input and returned source identity in the document manifest so a fragment variant does not masquerade as another source. A list of document texts without a manifest is difficult to reconcile when the original pages change.

Retain the collection state with the text

Static initial HTML is the scope. A page that assembles its content in a browser may yield little useful text even when the HTTP response succeeds. That does not prove that the website has no content. It tells you that this collector's observation scope may be insufficient.

The installed fetch is bounded to 600 KB, a twenty-second request timeout, and up to five redirects. Keep source gaps beside the Markdown. The current build does not propagate byte-cap truncation into a row.partial flag. Consequently, partial:false is not proof that the whole page was collected. Long-page completeness needs an independent check against the source rather than an assumption based on a successful conversion.

Separate successful observations, failed sources, blocked targets, and withheld delivery. An operational failure must not become an empty document that a later system interprets as negative evidence. At the same time, do not assume that an empty word count automatically makes a successful page observation free: the installed runtime marks successful HTTP analysis as found even when wordCount is zero.

Build a document manifest before an index

A useful manifest stores the requested URL, final observed source, observation time, content identity, collection state, and the location of the Markdown artifact. The exact storage layout is your application choice. The key is that someone should be able to move from a retrieved passage back to the particular source observation that supplied it.

Keep conversion and embedding as separate stages. A text-conversion success does not prove that chunking preserves context, that an embedding model retrieves the right passage, or that a generated answer cites it accurately. Evaluate those later steps using documents whose source and collection gaps remain visible.

For an acceptance exercise, choose a small static page, a long page expected to reach a collection bound, and a JavaScript-heavy page. Compare source scope and independently check the long page for truncation rather than relying only on a Markdown string or the partial flag. Add a repeated normalized URL and a failed target to inspect deduplication and failure routing. These are proposed checks, not newly executed cloud observations.

A retry should retain the earlier attempt's identity and outcome. If a remote delivery is uncertain, reconcile the saved run and Dataset before launching another job or appending another document. That avoids turning an operational recovery into duplicate corpus entries. Once collection is traceable, you can make a defensible decision about which documents are ready for retrieval and which require another source or human review.

Cost and source boundary

The live FREE-tier event prices checked on 20 October 2026 are $0.002 per result-found event and $0.005 per automatic Actor-start event. The start event is memory-scaled: one event per GB, with a minimum of one event. A free result outcome therefore does not make an entire run free. Other account tiers have their own rates; check the live pricing panel before execution. These prices describe the installed contract, not measured customer revenue or a new runtime test.

Inspect the Actor, input, and current pricing.

AI-assisted drafting; technical statements reviewed against the installed Actor source and current API metadata. Examples are illustrative unless explicitly identified as saved observations.