The head of your CSV is lying: how 9,291 invoice numbers almost vanished

Ahmed Abdeltawab — Sat, 15 Aug 2026 06:19:41 +0000

Real transaction data is never clean — and the worst part is that it looks clean. This is a short story from a real dataset (UCI Online Retail: 541,909 e-commerce transactions) about the quietest way to destroy data: silent type coercion. All numbers below come verbatim from an executed notebook.

The head looks perfect

Peek at the first rows of the file and InvoiceNo parses as clean integers — 100% parse rate, full confidence. Any type-inference step, mine included, would call it int64 and move on.

Measure the whole file instead of the head, and the number drops to ~98%. The other 2%: invoice numbers starting with "C" — which in this dataset marks a cancellation. Coerce the column to numeric and every one of them becomes NaN:

Invoice numbers destroyed by numeric coercion: 9,291

DextraLoaderWarning: load: ambiguous decision(s):
column 'InvoiceNo': ambiguous - float64 at parse_rate=0.98

An entire class of business events — silently gone. No exception, no crash. That's what makes coercion the quietest bug in data work: the pipeline succeeds.

Why those 9,291 rows matter

They are not noise. They are the returns side of the business: cancelled orders worth 8.4% of everything sold. Lose them and every revenue number downstream is quietly wrong.

One example of what they catch: the dataset's apparent #1 bestseller, "PAPER CRAFT, LITTLE BIRDIE" (168,470 GBP), is a phantom — a single 80,995-unit order entered at 09:15 and fully cancelled at 09:27 the same morning. Only the preserved cancellation rows expose it. The genuine bestseller is a cake stand.

The fix: identifiers are labels, not quantities

No library can know that "InvoiceNo" is an ID — that's domain knowledge. What a tool can do is disclose its guess and hand you a replayable plan you can correct:

naive, plan = dx.load(CSV_PATH, return_params=True)   # warns: ambiguous at 0.98

plan["columns"]["InvoiceNo"]["dtype"] = "object"      # invoices are labels
plan["columns"]["StockCode"]["dtype"] = "object"      # product codes too

df = dx.load(CSV_PATH, params=plan)                   # deterministic replay

Cancellation invoices preserved: 9,288
InvoiceNo values lost: 0

The correction is now documented, versionable code — not a mystery cell someone ran once.

Three habits this taught me

Never trust the head of a file. The first rows parse clean at 100%; the truth lives in the full-file parse rate. Measure every cell.
Declare identifiers explicitly. Invoice numbers, product codes, phone numbers, zip codes — they look numeric and they are not. One schema line prevents the whole class of bug.
Split, don't delete. The final frame here is three frames: sales (530,104 rows, 97.8%), returns (9,288, 1.7%), oddities (2,517, 0.5%). Deleted rows can't answer questions later; split rows can — the returns table is where the phantom bestseller was caught.

Try it

The full walkthrough — from messy CSV to a business decision, every step printing what it did and why — is published fully executed on Kaggle: Rescuing 9,291 invoices from coercion (pydextra).

It runs on pydextra (pip install pydextra, MIT), a small library I built around one idea: every function prints a one-line Decision: explaining what it did and why. To be transparent: it's a personal educational-practical project, not a replacement for pandas — pandas is the engine underneath.

GitHub: https://github.com/ahmedabdeltawab602-collab/dextra
Docs: https://ahmedabdeltawab602-collab.github.io/dextra/
Previous article — data leakage, same AUC, hidden lie: https://dev.to/ahmedabdeltawab/an-adversarial-review-found-11-real-defects-in-my-python-library-best-decision-i-made-2h85

Data: Chen, D. (2015). Online Retail. UCI Machine Learning Repository. DOI 10.24432/C5BW33, CC BY 4.0.

An adversarial review found 11 real defects in my Python library. Best decision I made.

Ahmed Abdeltawab — Tue, 11 Aug 2026 12:23:15 +0000

I built and published a Python library — then put it in front of an adversarial review whose only job was to break it. This is what it found, and what the library looks like now.

What pydextra is

pip install pydextra, then import dextra as dx. It's a small data-analysis library with one obsession: disclosure. Its 63 public functions — plus 5 scikit-learn-compatible wrappers, 68 public callables in all — share one flag vocabulary and:

print a one-line Decision: explaining what they did and why,
keep an audit trail on the DataFrame,
and, wherever statistics are learned from data, return a replayable params plan — fit on train, replay verbatim on test — so train/test leakage becomes hard to commit by accident.

One real call, verbatim from the leakage-safe pipeline notebook:

train_fe, params = dx.featpipe(train, steps=steps, return_params=True)

Decision: Fitted a 3-step featpipe pipeline (handle_missing -> encode -> scale);
33 new column(s) produced; combined params is a versioned, JSON-serialisable
artifact. Apply to held-out data with featpipe(df_test, params=...).

The part I'm most proud of isn't a feature

An adversarial external evaluation found 11 evidence-backed defects in the library. Version 0.6.0 closed all 11 — each one reproduced first by a failing "red" test, then fixed until green, with ~35 permanent regression tests and zero-warning exit gates. The original independent audit had scored an earlier version 82/100; the updated referee evaluation of 0.6.0 passed every agreed gate — 10/10 on a measurable definition fixed before the work started — with the remaining limitations listed openly in the repo.

Since 0.6.0 the API is frozen by policy: no new features, hotfixes only. What you evaluate today is what you run next year.

Three notebooks, real mess

All three are published fully executed on Kaggle:

Rescuing 9,291 invoice numbers from silent coercion — how numeric coercion silently eats identifiers, and how a disclosure-first loader catches it.
Two pipelines, same AUC (0.8465) — one is lying — the wrong protocol and the right one produce identical single-run AUC (0.8465); only 10 repetitions expose the real optimistic bias (+0.0261 ± 0.0210, wrong side higher in 9 of 10 runs). The full write-up: Same AUC, hidden leak, or run it in 5 minutes on Colab.
Egypt food prices 2010–2026: a fully-Arabic EDA — garlic +214.8% in one jump. Deliberately written in Arabic: technical Arabic data-science content is rare, and this is a small contribution against that gap.

Honest scope

It's a personal educational-practical project — not a pandas replacement, and not aimed at production-scale pipelines. pandas is the engine underneath; pydextra adds the disclosure layer.

GitHub: https://github.com/ahmedabdeltawab602-collab/dextra
PyPI: https://pypi.org/project/pydextra/
Docs: https://ahmedabdeltawab602-collab.github.io/dextra/

If the idea of functions that explain themselves appeals to you, a GitHub star helps others find it.

DEV Community: Ahmed Abdeltawab