Every scraping tutorial ends at response.text. Mine used to too — and then the storage layer ate three weekends, so here is the boring half, with receipts.
I run a small daily pipeline: it pulls public Telegram channel posts every morning, cleans them, and ships a bundle. Scale is tiny — a few hundred messages a day. The rule that shaped everything: a scraped row is a fact about the past, and facts don't get edited.
The schema that survived
Each row is append-only NDJSON, one object per line:
{"url": "https://t.me/s/channel/123", "fetched_at": "2026-10-13T07:00:12+08:00", "sha256": "9f2c…", "text": "…", "channel": "channel"}
Four fields, and every one of them exists because a bug made me add it:
-
url— the locator. A buyer, my future self, or a stranger can re-fetch it. -
fetched_at— the receipt of when the fact was true. Timestamp is a claim, not proof, and I price it accordingly. -
sha256— hash of the stored bytes, so a claim is checkable without trusting me. -
text— the payload, deliberately denormalized. No joins, no foreign keys, no migration ceremony.
What append-only bought me for free
De-duplication is a hash comparison, not a merge strategy. The morning run recomputes the hash for each URL; if the hash is already in today's window, the post is skipped. No UPSERT, no conflict resolution. Content changed? New hash, new row, both rows kept. History answered the "what changed" question before I programmed it.
Migrations are copy operations. Moving from one folder layout to another was cat old/*.ndjson > new.ndjson. No schema, no dump/restore, no pgloader incident. With a few hundred rows a day, SQLite would also work fine — I chose flat files because the artifact is the product, and the artifact should not need a database engine to open.
Backups are rsync. No dump format to test, no restore drill that I keep postponing. The restore drill is "files exist, open them."
What it costs (the honest section)
- Queries are scans. Any "find me X" is a grep over the month's files, ~40 MB. Fine at my scale, embarrassing at a gigabyte. The exit ramp is DuckDB reading the same NDJSON — one line, no re-ingestion pipeline.
- Nothing enforces the schema. A typo in a field key lands silently in a line of JSON. My mitigation is a lint step in the cron job: every row must have the four keys; fail loud, alert, don't ship a dirty bundle.
- Timestamps are lies you're legally required to keep. The row proves the bytes matched the hash at some point, not when. I ship this caveat with the data, priced in.
The decision rule
If your scraped rows are facts you append and never edit, start with NDJSON and a hash — you get dedup, diffs, history, and backups for the price of a text format. The moment you need updates, joins, or sub-second queries at scale, graduate — and your graduates keep working, because DuckDB and pandas read these files natively.
Storage is not the sexy half of scraping. It is the half that decides whether next month's you trusts this month's data.
— 何雨禾 (heyuhe). I run this pipeline daily and sell the bundles; the same hashes and receipts described above ship with every row.
When someone asks you to prove where your data came from — a client, an audit, a compliance box you have to tick by a deadline — the provenance is the product; the rows are just the carrier. That is exactly what ships in my bundles: the append-only files above, with provenance header, per-row sha256 and fetched_at. Grab the receipts first, pay only if they'd save you an afternoon: free Telegram OSINT sample (suggested $5) · tg-crawl Pro: the scraper wheel + daily-crawl recipes ($9). Picking a format before your pipeline bends: CSV, JSONL, or SQLite.
One question, no pitch: if my daily Telegram bundle disappeared tomorrow, would you be very disappointed, somewhat disappointed, or not bothered? Reply with one word — I use it to decide whether to keep building it.
Top comments (0)