Freshness Watermarks Belong on Every Row, Not the README

The most common freshness design I see — in my own old code included — is a single stamp at the top: a README that says "last updated 2026-10-01", a dashboard header that says "data as of Monday", a manifest with one generated_at. It is one adjective covering a million rows, and it is the exact same lie as the green LIVE badge, just moved up the stack.

Here is the part the README-banner design hides: a dataset is never one age. My daily Telegram dump is 340 rows, and on a normal day it contains rows observed this morning, rows last confirmed four days ago because the channel went quiet, and a reference feed that legitimately updates monthly. One banner timestamp can only describe one of those three honestly, and whichever one it names is a lie about the other two. The banner says "as of 08:00" and the reader correctly infers everything is fresh; the quarterly registry row is nine weeks old and the banner just laundered it.

The rule I now enforce, and it's the direct sequel to the runbook-rot argument: a document gets one freshness stamp; a dataset gets one per row. The runbook post applies the same math to prose — one as_of per doc, lag against cadence, CI fails the build — and the doc layer is honestly the easy case, because one stale runbook costs one outage. In a dataset, stale rows are silent, individually plausible, and consumed by code, not humans. One stale row costs one wrong query, one wrong chart, one wrong decision, with no 3 a.m. phone call.

The three columns, and what each one is allowed to mean

Every row in my pipeline now carries three timestamps, and the discipline is entirely in what each is forbidden to mean:

{
  "url": "https://t.me/s/example/12345",
  "observed_at": "2026-10-18T07:59:12+08:00",
  "ingested_at": "2026-10-18T08:02:41+08:00",
  "as_of":       "2026-10-18T07:59:12+08:00",
  "content_sha256": "9f3a…"
}

• observed_at — the last time the source confirmed this record. For a scraped post, the publish/edit time the page itself reports, or the fetch time when the page gives nothing. This is the only column freshness queries may read.
• ingested_at — the last time my pipeline touched bytes for this row. Backfills move this column and are forbidden from touching the other two.
• as_of — the freshness claim, materialized so consumers don't recompute it. Rule: as_of = observed_at, never now(). The column exists as a tripwire: any ETL that writes as_of = now() on replay is visible in a one-line diff against observed_at.

Two columns where one would do is the whole point. Replay laundering — a backfill rewriting timestamps so nine-day-old facts look fresh — is invisible in single-timestamp schemas and trivially detectable in two:

rows = [r for r in dataset if r["ingested_at"] - r["observed_at"] > MAX_HONEST_LAG]
for r in rows:
    print(f"SUSPECT: ingested {r['ingested_at']}, source last confirmed {r['observed_at']}")

That audit is the "diff the two columns" check from the watermark pattern, run per row. It finds the laundered rows by name, not by suspicion.

Per-row makes per-source alarms possible; the README banner makes them impossible

The watermark math needs rows to work:

watermark(feed) = max(as_of) over rows in feed
lag(feed)       = now() - watermark(feed)
alarm(feed)     = lag(feed) > k × expected_interval(feed)   # k ≈ 3

With a README banner you get one as_of for the whole file, which means one watermark, which means the loudest feed in your dataset silences every quiet one. My hourly channel updates at 09:00, my quarterly registry last confirmed in July: a file-level watermark computes 09:00, and the registry's nine-week lag never crosses any threshold, because the aggregate math can't see it. The banner design doesn't fail to alert on the registry; it structurally hides the registry. Partial death — one of twelve feeds stops, aggregate row counts wobble inside normal noise — is invisible to aggregate freshness and obvious to per-row freshness the same day.

The export version of this is what sells it for me, because data products leave the building. Every bundle I ship opens with a manifest that lists per-source as_of tables, not a single generation timestamp:

channel          rows  as_of (max)          expected_interval  lag
alerts-daily     118   2026-10-18T07:59+08  1h                 3m
market-watch      96   2026-10-18T06:12+08  2h                 2h11m
registry-ref      12   2026-08-30T00:00+00  quarterly          45d

Three feeds, three honest ages, one artifact. A buyer — or their compliance team, which is who actually asks — can see exactly which rows are current and which are historical, inside the file they downloaded, without trusting my dashboard, my README, or me. A data product whose only freshness statement is "generated 2026-10-18" is a rumor with good formatting; a data product with per-row as_of is an invoice.

The cost, stated honestly

Per-row freshness is not free, and the bill is boring: one extra column at write time (the fetch already knows observed_at; you're persisting what you already had), one audit job, and one rule enforced in code review — replays move ingested_at only. The README-banner design costs less because it answers a question nobody needs answered ("when did you last regenerate?") and dodges the one question that matters ("how old is the row I'm reading right now?").

If you keep one habit from the runbook post and this one, it's a single sentence: freshness is a property of the smallest unit your reader consumes. For a runbook, that unit is the document, so the doc gets an as_of and CI guards it. For a dataset, the unit is the row, so every row carries its own age, and the README gets one line: "see as_of per row; per-source watermarks in the manifest." The banner isn't dead — it's demoted to a pointer.

Rows lie in aggregate and tell the truth individually. Stamp them accordingly.

— 何雨禾 (heyuhe). Every row I sell carries its own as_of for the selfish reason that a buyer's audit is my invoice's bodyguard.

---

Every row in my bundles ships with observed_at / ingested_at / as_of and the per-source watermark table above — ages you can check before you trust them: free Telegram OSINT sample (suggested $5) · Premium 2-day dump, 6 channels ($30) · tg-crawl Pro: the scraper wheel + daily-crawl recipes ($9) · daily freshness report ($5). The prose version of this rule (one as_of per runbook, CI-fails-the-build): Runbook Rot: Three Fingerprints of Stale CI Docs.