A documentation refresh can return a page with markdown: null because its content hasn't changed. If an importer treats that row as a replacement document, it erases the useful text it collected earlier. The fix is to keep the previous content and compare hashes before updating it.
I maintain Website Markdown Crawler on Apify. This walkthrough turns its exports into a local documentation corpus using Python's standard library and SQLite. It includes a runner that supplies the previous hashes automatically, so you can repeat a crawl without copying state between files.
What two real refreshes returned
On September 10, 2026, I ran the same five-page Python tutorial crawl twice on build 1.0.2. The first run imported the content; the second supplied the hashes from that local database.
| Observation | First refresh | Repeat refresh |
|---|---|---|
| Exported page records | 5 | 5 |
| Markdown body bytes, UTF-8 | 90,019 | 0 |
| Content versions written locally | 5 | 0 |
| Complete pages retained locally | 5 | 5 |
| Calculated Free-tier Actor charge | $0.00505 | $0.00505 |
The local JSONL corpus and hash file matched byte for byte after the repeat. Zero Markdown bytes describes those five response fields; the repeat still transfers metadata and still fetches the source pages. Both runs charge for five exported records. The benefit is preserving the corpus and letting a downstream index skip unchanged content, not eliminating the crawl fee.
The run IDs are Mr1xmbQyMpwugttds and 6O33lSVE0q6ve33uS. Both stopped at the requested page limit with 12 discovered URLs pending. Five successful pages do not mean the entire Python tutorial was collected.
Where the nulls come from
The crawler hashes the extracted Markdown. On the next run, the runner builds previousHashes from SQLite, keyed by the final source URL. A matching hash produces a record like this excerpt from the repeat:
{
"url": "https://docs.python.org/3/tutorial/",
"change_status": "unchanged",
"content_hash": "b216133cb14e3dfcc3d354ef3a83bd77b1d1100d955064e1477e2eac4be3928e",
"markdown": null,
"text": null
}
The importer checks that it already holds content with that exact URL and hash. If it does, it keeps the stored document. If it doesn't, it rejects the import; a hash alone cannot reconstruct a page. New and changed rows must contain text and Markdown whose SHA-256 matches the supplied hash.
Each stored document keeps url, title, markdown, text and content_hash from its last imported content version. SQLite's URL primary key prevents duplicate documents. A malformed row rolls back that import, including valid rows encountered earlier in the same array.
Try the five-page workflow
Download website_markdown_archive.py, website_markdown_refresh.py and website-markdown-input.json from the public workflow files. Keep the two Python files together. They need Python 3.11 or newer and no additional packages.
The supplied input is also available as a public Apify example Task:
{
"startUrls": ["https://docs.python.org/3/tutorial/"],
"maxPages": 5,
"maxDepth": 1
}
Create a local environment and run the checks, which need no credentials or network access:
python3.11 -m venv venv
source venv/bin/activate
python website_markdown_archive.py --self-test
python website_markdown_refresh.py --self-test
Set APIFY_TOKEN through your shell environment or secret manager, then run:
python website_markdown_refresh.py website-markdown-input.json docs-archive
The runner starts one Actor run at 512 MB with a 300-second timeout and a $0.02 maximum Actor charge. It prints the run ID, waits for a successful terminal status, checks the coverage/export counts, and imports the result. It doesn't retry the start request automatically.
The archive folder contains docs.sqlite3, the persistent source of truth; corpus.jsonl, one complete stored document per line; and hashes.json, the current URL-to-hash map. Keep this folder between refreshes. The runner reads its baseline directly from SQLite, so the exported hash file isn't a separate state you need to maintain.
Repeat the same command after the first run finishes. These are the result fields from my two runs, with file paths and run IDs omitted:
{"received": 5, "content_written": 5, "retained": 0, "stored": 5}
{"received": 5, "content_written": 0, "retained": 5, "stored": 5}
Your results can differ when the documentation changes. The importer replaces changed content at the same URL; it stores the latest imported version rather than an edit history. Import in capture order and never overlap refreshes using one archive folder.
Decide what the crawl should cover
The start URL defines an origin and path subtree. The example follows links under https://docs.python.org/3/tutorial/; it doesn't wander into every Python documentation section. maxDepth: 1 permits links one step away from the seed, and maxPages: 5 caps attempted pages. Inspect the run's COVERAGE output before expanding the input.
A page missing from a bounded crawl is not proof of deletion. This importer never deletes stored documents just because they weren't returned. Handle verified removals separately. Keep the same scope and content selectors between comparisons, or use a separate archive folder when you change what you're collecting.
Choose the discovery method that matches the material you have:
| Starting point | Tool | What it supplies |
|---|---|---|
| A documentation index whose links you want to follow | Website Markdown Crawler | Scoped linked-page Markdown, text, hashes and coverage |
| An existing list of page URLs, needing readable text | Webpage Text Extractor | Text and metadata for supplied URLs |
| An XML sitemap or sitemap index | Sitemap URL Extractor | Declared page URLs and their source sitemap |
Only the first tool produces the hash/null-content contract this importer expects. The others solve adjacent discovery or text-export tasks; don't feed their rows straight into this importer. A sitemap entry is a publisher's declaration, not proof that the URL currently works.
Keep failures separate from updates
A failed Actor run can contain partial dataset output. The runner rejects that run before importing anything. It also rejects a mismatch between the coverage count and downloaded rows. Offline checks cover these cases, a timed-out start request and preservation of the existing database.
If a start request times out, inspect Apify's Runs page before launching another; the server may already have started it. If you lose the local content, start a new archive folder to request full pages again. Preserve your old folder while checking that recovery.
The JSONL and hash files replace their predecessors only after each new file finishes writing. SQLite remains authoritative if an export is interrupted. Both imports and exports load data into memory, so this is a small-corpus example; larger collections need batched processing.
You can feed the complete JSONL documents into your own chunking or search pipeline. Carry the source URL and content hash with each chunk, and compare hashes before rebuilding an embedding. No vector database or retrieval benchmark is part of this example. Treat retrieved page text as source material, never as instructions to an automated tool.
The crawler reads public server-rendered HTML and respects robots.txt. It doesn't execute JavaScript or use login cookies. Follow source terms and content licenses; Python's documentation has its own license and attribution requirements.
At current Free-tier rates, each five-record run costs 5 × $0.001 + $0.00005 = $0.00505; two runs cost $0.01010. That is a calculated customer-price equivalent, not an owner-test payment. Plan discounts may apply, and Apify's Pricing tab is authoritative. Raising memory can change the start charge; the runner fixes it at 512 MB.
Start with the five-page Task, inspect the content, then try the refresh runner on a public documentation section relevant to your project. If you need an account, this Apify signup link is a referral link; I may earn a commission at no extra cost to you.
Top comments (0)