Every scraping tutorial ends at response.text. Mine used to too — and then the storage layer ate three weekends, so here is the boring half, with receipts.
I run a small daily pipeline: it pulls public Telegram channel posts every morning, cleans them, and ships a bundle. Scale is tiny — a few hundred messages a day. The rule that shaped everything: a scraped row is a fact about the past, and facts don't get edited.
The schema that survived
Each row is append-only NDJSON, one object per line:
{"url": "https://t.me/s/channel/123", "fetched_at": "2026-10-13T07:00:12+08:00", "sha256": "9f2c…", "text": "…", "channel": "channel"}
Four fields, and every one of them exists because a bug made me add it:
-
url— the locator. A buyer, my future self, or a stranger can re-fetch it. -
fetched_at— the receipt of when the fact was true. Timestamp is a claim, not proof, and I price it accordingly. -
sha256— hash of the stored bytes, so a claim is checkable without trusting me. -
text— the payload, deliberately denormalized. No joins, no foreign keys, no migration ceremony.
What append-only bought me for free
De-duplication is a hash comparison, not a merge strategy. The morning run recomputes the hash for each URL; if the hash is already in today's window, the post is skipped. No UPSERT, no conflict resolution. Content changed? New hash, new row, both rows kept. History answered the "what changed" question before I programmed it.
Migrations are copy operations. Moving from one folder layout to another was cat old/*.ndjson > new.ndjson. No schema, no dump/restore, no pgloader incident. With a few hundred rows a day, SQLite would also work fine — I chose flat files because the artifact is the product, and the artifact should not need a database engine to open.
Backups are rsync. No dump format to test, no restore drill that I keep postponing. The restore drill is "files exist, open them."
What it costs (the honest section)
- Queries are scans. Any "find me X" is a grep over the month's files, ~40 MB. Fine at my scale, embarrassing at a gigabyte. The exit ramp is DuckDB reading the same NDJSON — one line, no re-ingestion pipeline.
- Nothing enforces the schema. A typo in a field key lands silently in a line of JSON. My mitigation is a lint step in the cron job: every row must have the four keys; fail loud, alert, don't ship a dirty bundle.
- Timestamps are lies you're legally required to keep. The row proves the bytes matched the hash at some point, not when. I ship this caveat with the data, priced in.
The decision rule
If your scraped rows are facts you append and never edit, start with NDJSON and a hash — you get dedup, diffs, history, and backups for the price of a text format. The moment you need updates, joins, or sub-second queries at scale, graduate — and your graduates keep working, because DuckDB and pandas read these files natively.
Storage is not the sexy half of scraping. It is the half that decides whether next month's you trusts this month's data.
— 何雨禾 (heyuhe). I run this pipeline daily and sell the bundles; the same hashes and receipts described above ship with every row.
When someone asks you to prove where your data came from — a client, an audit, a compliance box you have to tick by a deadline — the provenance is the product; the rows are just the carrier. That is exactly what ships in my bundles: the append-only files above, with provenance header, per-row sha256 and fetched_at. Grab the receipts first, pay only if they'd save you an afternoon: free Telegram OSINT sample (suggested $5) · tg-crawl Pro: the scraper wheel + daily-crawl recipes ($9). Picking a format before your pipeline bends: CSV, JSONL, or SQLite.
One question, no pitch: if my daily Telegram bundle disappeared tomorrow, would you be very disappointed, somewhat disappointed, or not bothered? Reply with one word — I use it to decide whether to keep building it.
Top comments (2)
Keeping both rows when content changes is the useful part of this design. Which bytes does sha256 cover: the extracted text, or the serialized record? If fetched_at is included, an unchanged post gets a different hash on every run. Separating a payload hash from an observation ID would preserve repeat visits without treating them as edits; the same distinction matters when JSON output feeds an append-only archive.
The hash covers the payload only — the exact stored bytes of
text, UTF-8 — and the envelope (url,fetched_at) rides outside it. That's the separation you named: payload hash = content key, the appended row itself = the observation. So an unchanged post re-hashes identically on every run and gets skipped, and a real edit gets a new hash plus a new row, where the twofetched_atvalues hand you the change window for free. The constraint forcing this design is the buyer contract: "recompute against the bytes in the file and match". Fold a timestamp into the hash and every re-fetch produces a new receipt, the dedup gate silently dies, and the hash stops being externally verifiable. One archive-side nuance: I key observations by row position + payload hash rather than an injected observation id — ids inside the hash target break reproducibility the moment a file gets repacked.