DEV Community

Yuhe He
Yuhe He

Posted on

Your Dataset's README Is Part of the Data - Audit It Before Every Publish

#data #documentation #webscraping

Every pipeline has two outputs: the data, and the claims you make about the data. Most teams version the first and vibe about the second. I learned this the expensive way while publishing a small public dataset this week.

The label that snuck in

The dataset was a sample of public Telegram messages collected by a scraper that takes timestamped snapshots of channels. The README described it as "keyword-filtered, hand-checked, deduped."

Then I ran the pre-publish audit I now run on everything I ship, and the README lost:

  • keyword-filtered — the export pipeline has no keyword step. The keyword filtering happens during collection, upstream, on a machine that no longer exists. The file is a raw slice.
  • hand-checked — true for a subset, not the whole file. A claim with hidden scope is a lie with extra steps.
  • deduped — true by message ID, false by content. Clone rings paste identical text across channels, so the file contains near-identical rows on purpose. They are the finding, not a defect.

Two of three claims were wrong, and none of them lived in the CSV. They lived in the prose, where no linter runs.

Why this matters more than a typo

A consumer grades your target using your labels. If someone filters my "keyword-filtered" sample expecting keyword density and finds raw noise, they don't conclude "the README was loose" — they conclude the dataset is junk, and often the whole author is junk. The label is load-bearing.

The same failure shows up in scoring tools: a secret scanner that screams CRITICAL at a JavaScript polyfill's empty password="" string is grading its own parser, not the target. Your README is a scanner too. Audit its eyes.

The ten-line gate

The fix is not a style guide. It is a gate that runs at publish time, against the actual artifact:

import csv, json, sys

def audit(csv_path, declared):
    with open(csv_path) as f:
        rows = list(csv.DictReader(f))
    cols = set(rows[0].keys()) if rows else set()
    problems = []
    for name, kind in declared.items():
        if kind == "column" and name not in cols:
            problems.append(f"README declares column '{name}'; export has {sorted(cols)}")
    for col in cols:
        vals = [r[col] for r in rows]
        if all(v == "" for v in vals):
            problems.append(f"column '{col}' exists but is 100% empty")
    return problems
Enter fullscreen mode Exit fullscreen mode

Declare each README claim as something checkable — a column that must exist, a row count range, a "no column is fully empty" invariant, a dedupe key that must be unique — and fail the publish when prose and bytes disagree. My gate now runs before every dataset version push, the same way tests run before a deploy. It caught both label bugs above before they shipped; the third ("deduped") was reworded to "deduped by message ID" because the near-duplicates are the point.

The rule

Treat the README as a build artifact, not marketing copy. Regenerate it, or at minimum diff it, at publish time. If the prose was written for a previous version of the pipeline, it describes data that no longer exists — and you are the scam with good formatting.

What claims in your README would fail a diff against your own export?

Free tooling from the same pipeline: Telegram OSINT bundle sample (PWYW) · weekly digest $15 · methodology posts free here on dev.to.

Top comments (0)