Silent Row Loss: A Pinned Adversarial Corpus for Import Pipelines

A broken import rarely crashes. It reports success, drops eleven rows, and ships a payroll file with three people missing. The durable fix is not a smarter parser. It is a corpus that pins what the parser must survive, plus a gate that fails when coverage shrinks.

That asset matters more each quarter. Generated code now produces most one-off CSV and JSON ingest scripts, and generated scripts come with generated assumptions. The corpus is the part a human still has to own.

Partial success is the whole problem

Import bugs cluster into four shapes. None of them raise an exception on their own.

• Row loss. A quoted newline splits one record into two, and the parser silently keeps the valid half.
• Type coercion. 1.234,56, $1,234.56, and 1234.56 reach the same column, and one becomes null.
• Order dependency. The mapper reads by index instead of by header name, so a reordered export breaks amounts.
• Encoding drift. A BOM lands inside the first header, and employee_id becomes \ufeffemployee_id.

Every one of these passes a happy-path unit test written from the same assumptions as the code. That is the trap: tests authored next to the parser inherit the parser's blind spots.

Pin intent, not bytes

A corpus without expectations is a folder of junk. Each fixture needs a manifest that states what must happen, so a future reviewer can disagree with a decision instead of guessing it.

{
  "id": "csv-0007-crlf-quoted-newline",
  "why": "Mail clients wrap long address lines inside quoted fields; a real export did this in 2025-08",
  "origin": "model-draft+human-review",
  "file": "csv-0007.csv",
  "sha256": "9f2c0f1b7d5c4a3e8b6d0c2a4f5e7d9b1c3a5e7f9b0d2c4a6e8f0b1d3c5a7e9f",
  "expect": {
    "accepted_rows": 4,
    "rejected_rows": 0,
    "warning_codes": ["QUOTED_NEWLINE"]
  }
}

Four fields carry the weight. why forces a realistic story, not a fuzzed byte. expect is exact, including warning codes. sha256 makes byte edits visible in review. origin records whether a human read the trap before it landed.

Rules that keep the corpus honest:

1. No fixture without a why of at least 20 characters.
2. No fixture without an exact expect, including the warning list.
3. Any byte edit re-pins the hash and re-states the expectation in the same commit.
4. Duplicate ids fail the run, because duplicated intent hides coverage gaps.

Forge traps fast, then distrust all of them

Writing thirty realistic traps by hand is a week of tedium. Drafting them is a good fit for a cheap model. Bulk, low-stakes, easy to falsify by a single run of the gate.

The prompt skeleton below produced the useful drafts. It is deliberately narrow.

Generate 20 adversarial CSV fixtures for a payroll importer.
Constraints:
- one file per fixture, UTF-8 bytes, at most 8 data rows
- synthetic values only: no real names, no real IBANs, no real addresses
- each fixture targets exactly one trap:
  quoted newline, embedded delimiter, BOM, CRLF, trailing empty column,
  semicolon delimiter, currency symbol, decimal comma, duplicate header,
  header case change, blank line in the middle, formula-like leading =
- return filename, the raw bytes as a base64 block, and one "why" line
- no fixture may require a network call or an external file

MonkeyCode is an open-source project that offers free model access, which the operator describes as a 10-million-token allowance. That is enough volume for this kind of bulk drafting without a budget approval. Check the project page for current limits before sizing a team workflow on them.

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

Acceptance rate stayed low, and that is the point. Most drafts were duplicates or unrealistic. Each surviving draft needed three human decisions: is this trap realistic, what is the exact expectation, and which warning code should fire. A model that proposes traps is useful. A model that decides expectations is a liability.

A runner small enough to read in one pass

The runner does not test the parser. It tests the corpus against the parser, and it fails loudly when the two disagree.

# corpus_runner.py — stdlib only
import hashlib, json, pathlib, sys

from importer import import_csv          # adapter: path -> Report

CORPUS = pathlib.Path("fixtures")
REQUIRED = {"id", "why", "file", "sha256", "expect"}


def manifests():
    for path in sorted(CORPUS.glob("*.json")):
        yield json.loads(path.read_text(encoding="utf-8"))


def check(meta, seen):
    problems = []
    missing = REQUIRED - meta.keys()
    if missing:
        return [f"{meta.get('id', '?')}: missing keys {sorted(missing)}"]
    if meta["id"] in seen:
        problems.append(f"{meta['id']}: duplicate id")
    if len(meta["why"]) < 20:
        problems.append(f"{meta['id']}: 'why' too thin to review")
    fixture = CORPUS / meta["file"]
    if not fixture.exists():
        return problems + [f"{meta['id']}: fixture file missing"]
    if hashlib.sha256(fixture.read_bytes()).hexdigest() != meta["sha256"]:
        problems.append(f"{meta['id']}: bytes changed, re-pin hash and expect")
    report = import_csv(str(fixture))
    want = meta["expect"]
    if report.accepted != want["accepted_rows"]:
        problems.append(
            f"{meta['id']}: accepted {report.accepted}, want {want['accepted_rows']}")
    if report.rejected != want["rejected_rows"]:
        problems.append(
            f"{meta['id']}: rejected {report.rejected}, want {want['rejected_rows']}")
    got = sorted(report.warning_codes)
    if got != sorted(want["warning_codes"]):
        problems.append(f"{meta['id']}: warnings {got}, want {sorted(want['warning_codes'])}")
    return problems


def main():
    seen, failures, codes = set(), [], set()
    for meta in manifests():
        failures += check(meta, seen)
        seen.add(meta.get("id"))
        codes.update(meta.get("expect", {}).get("warning_codes", []))
    if len(codes) < 5:
        failures.append(f"corpus covers {len(codes)} warning codes, gate is 5")
    for line in failures:
        print(f"FAIL {line}")
    print(f"{len(seen)} fixtures, {len(failures)} failures")
    return 1 if failures else 0


if __name__ == "__main__":
    sys.exit(main())

The import_csv adapter is one function, and it should stay one function. If the corpus needs the whole application to boot, the suite dies within a month. Three behaviours the runner depends on: a Report with accepted, rejected, and warning_codes.

A coverage gate sits at the bottom. Five distinct warning codes is an arbitrary floor, and it is a floor that catches the real regression: a team deletes three fixtures during a refactor and quietly loses a category of trap.

Permutation beats another example

One property is worth more than ten extra fixtures. Column order must not change the result.

# test_permutation.py
import csv, random, tempfile
from pathlib import Path

from importer import import_csv


def rewrite(src: Path, order, dst: Path):
    with src.open(newline="", encoding="utf-8-sig") as fin:
        rows = list(csv.reader(fin))
    header, body = rows[0], rows[1:]
    idx = [header.index(name) for name in order]
    with dst.open("w", newline="", encoding="utf-8") as fout:
        w = csv.writer(fout)
        w.writerow([header[i] for i in idx])
        for row in body:
            w.writerow([row[i] for i in idx])


def test_column_order_is_irrelevant():
    src = Path("fixtures/csv-0002-plain.csv")
    with src.open(newline="", encoding="utf-8-sig") as f:
        header = next(csv.reader(f))
    base = import_csv(str(src))
    rng = random.Random(20261107)
    for _ in range(10):
        order = header[:]
        rng.shuffle(order)
        tmp = Path(tempfile.mkdtemp()) / "perm.csv"
        rewrite(src, order, tmp)
        got = import_csv(str(tmp))
        assert got.accepted == base.accepted, order
        assert sorted(got.warning_codes) == sorted(base.warning_codes), order

One caveat is built into the helper. csv.writer normalises quoting, so a warning that depends on the original quoting style can shift after the rewrite. A failure here means one of two things: the mapper is order-sensitive, or the rewrite changed the bytes it was measuring. Both are findings worth a ticket.

Triage: importer bug or expectation bug

The harder daily question is not "did it fail" but "who is wrong". A fixed table beats a debate in a pull request thread.

Symptom	Most likely cause	First action	Quarantine allowed
Accepted count drops after a parser change	Parser regression	Bisect the adapter, revert first	No
Warnings differ only by order	Expectation written as a list, not a set	Fix the comparison, not the parser	No
Hash mismatch on CI	Fixture bytes edited outside review	Re-pin hash and expectation in the same commit	No
Importer is right, expectation was wrong	Reviewer misread the export format	Rewrite why and expect, note it in the commit	Yes, one commit
Trap now impossible to produce	Format is rejected earlier in the pipeline	Delete fixture and record the reason	Yes, permanent

Quarantine means a skip list in the runner with a linked issue and an owner. It never means deleting the manifest. A quiet deletion is how a corpus loses its memory.

Hosting the runner without growing a liability

Replay matters outside the team. Support engineers want to re-run a customer's failing file without a laptop setup. The operator also offers a free server option, which is enough to host the corpus runner at a stable URL for exactly that scenario.

Boundaries for that deployment:

1. Fixtures stay synthetic. No customer file enters the shared instance, ever.
2. Uploads go to a scratch path with a TTL. The runner reads, reports, and deletes on a timer, not on trust.
3. No production credentials. The runner needs read access to fixtures and nothing else.
4. Expect recycling. Free instances restart without notice, so treat local state as disposable and keep fixtures in git.
5. Fail visible. If the instance is down, the CI gate must run the corpus locally instead of skipping it.

A shared runner on a free tier is a convenience. It is not a trust boundary, and it should never become the only place the corpus can execute.

Who should skip this

The approach fits teams that own a recurring import path with a stable adapter seam. It does not fit everyone.

• Teams processing regulated personal data need a reviewed fixture policy before any corpus lands in git.
• Teams without a single entry function will spend more time building the adapter than writing fixtures.
• Teams whose imports are already covered by contract tests against a partner schema should extend those tests, not fork a second corpus.
• Teams that cannot name an owner for the CI gate should keep the corpus local, because an unowned gate gets disabled in month two.

The corpus also rots if nobody prunes it. Two fixtures describing the same trap, or six fixtures guarding a format the vendor retired, both create noise that trains reviewers to ignore failures.

Start with one file that already hurt

Pick a file a customer sent last quarter. The one that produced a support ticket, or a spreadsheet someone fixed by hand at 6pm.

Run it through the adapter. Write down the accepted count, the rejected count, and every warning. That is the first manifest, and it costs one hour.

Then add the second file: the one nobody has sent yet, but the format allows. Two manifests, one runnable gate, and a rule that no parser change ships without a hash-pinned expectation behind it. The corpus grows from there, one real failure at a time.