Keep 0017 as text: a Python CSV checker for product imports

Igor Spravtsev — Fri, 16 Oct 2026 14:09:04 +0000

The Python CSV checker below is a small script that validates a supplier's product file before import. It keeps model codes as text and reports records that break the agreed format or repeat a product key. Hold the entire import on either kind of problem.

Suppose a supplier sends an equipment catalogue update. Model 0017 has one document; model 17 has another. An importer that converts both codes to integers now has one key: 17. The CSV can parse successfully while the next step attaches a document to the wrong product.

The Python CSV checker below reads the received file and prints a JSON report with counts, record numbers, and problems to review. It never edits the source or writes to a database.

Start with a file you can inspect

Save this as catalog.csv. All manufacturers, model codes, and document names in this tutorial are invented. The examples are small enough to check by eye.

maker_id,model,document
M01,0017,P-0017.pdf
M01,17,P-17.pdf
M01,AX-5,P-AX5.pdf
M01,АX-5,P-AX5-alt.pdf
M02,AX-5,Q-AX5.pdf
M01,AX-5,P-AX5-rev2.pdf
M01,AX-5 ,P-AX5.pdf

There are seven data records. Two details deserve a closer look:

0017 and 17 are different model codes in this example. A leading zero is part of the identifier.
The first character of АX-5 in record 5 is Cyrillic А, U+0410. The first character of AX-5 in record 4 is Latin A, U+0041. They look alike in many fonts.

Record numbers here include the header as record 1. They count CSV records, which can span physical lines when a field is quoted. The report deliberately calls the field record rather than line.

Agree what a valid product key looks like

For this demo, the supplier contract is deliberately narrow:

Field	Agreement
`maker_id`	An assigned manufacturer identifier using uppercase ASCII letters and digits
`model`	Uppercase ASCII letters and digits, optionally separated by `.`, `/`, or `-`
`document`	A nonempty reference; filename syntax and document contents are not checked
File	Comma-separated UTF-8, with an optional UTF-8 BOM and the exact header shown above

No field may have surrounding whitespace or ASCII control characters. The unique key is the exact pair (maker_id, model). M01 / AX-5 and M02 / AX-5 are therefore separate products.

These are rules for this example, not rules for every equipment catalogue. If a supplier uses Cyrillic codes, lowercase letters, or meaningful spaces, write a contract that accepts those values. Rejecting Cyrillic here signals a disagreement with this particular contract. It says nothing about whether that alphabet is valid in general.

Avoid a repair such as padding every code to four digits. That would turn 17 into 0017, merging two identifiers that the source treats separately. Replacing a similar-looking character needs the same care: ask the source owner which code was intended.

Run the Python CSV checker

Python's csv.reader returns strings by default. The reader below does not request numeric conversion. There is no call to int() on a model code.

The file is opened with newline="", as the CSV documentation recommends. The utf-8-sig codec accepts UTF-8 with or without a starting BOM. It does not guess other encodings. A semicolon-separated file also fails this contract rather than being silently reinterpreted.

Save the complete checker as check_catalog.py. It uses only the Python standard library; the results below were run with Python 3.13.0.

#!/usr/bin/env python3
"""Validate a small supplier CSV before joining it to a product catalogue.

Demo contract: comma-separated UTF-8, exact header, uppercase ASCII codes.
The input is never changed. Exit 0: pass; 1: rejected records; 2: unreadable input.
"""

import argparse
import csv
import json
import re
import sys


HEADER = ["maker_id", "model", "document"]
MAKER = re.compile(r"[A-Z0-9]+")
MODEL = re.compile(r"[A-Z0-9]+(?:[./-][A-Z0-9]+)*")


def check(path):
    issues = []
    seen = {}
    total = 0
    with open(path, encoding="utf-8-sig", newline="") as source:
        reader = csv.reader(source, strict=True)
        if next(reader, None) != HEADER:
            raise ValueError("expected header: maker_id,model,document")
        for record, values in enumerate(reader, start=2):
            total += 1
            problems = []
            if len(values) != len(HEADER):
                problems.append(f"expected 3 fields, got {len(values)}")
            else:
                item = dict(zip(HEADER, values))
                for name, value in item.items():
                    if not value:
                        problems.append(f"{name}: empty")
                    elif value != value.strip():
                        problems.append(f"{name}: surrounding whitespace")
                    if any(ord(c) < 32 or ord(c) == 127 for c in value):
                        problems.append(f"{name}: control character")
                if not MAKER.fullmatch(item["maker_id"]):
                    problems.append("maker_id: expected uppercase ASCII letters/digits")
                if not MODEL.fullmatch(item["model"]):
                    extra = " ".join(
                        f"U+{ord(c):04X}" for c in item["model"] if not c.isascii()
                    )
                    problems.append("model: outside the agreed grammar" +
                                    (f" ({extra})" if extra else ""))
                if not problems:
                    key = (item["maker_id"], item["model"])
                    if key in seen:
                        previous, document = seen[key]
                        detail = ("same document" if document == item["document"]
                                  else "different document")
                        problems.append(f"key repeats record {previous}: {detail}")
                    else:
                        seen[key] = (record, item["document"])
            if problems:
                issues.append({"record": record, "problems": problems})
    return {"records": total, "rejected_records": len(issues), "issues": issues}


def main():
    parser = argparse.ArgumentParser(description=__doc__)
    parser.add_argument("csv_file")
    args = parser.parse_args()
    try:
        result = check(args.csv_file)
    except (OSError, UnicodeError, csv.Error, ValueError) as exc:
        print(json.dumps({"input_error": str(exc)}, ensure_ascii=True), file=sys.stderr)
        return 2
    print(json.dumps(result, ensure_ascii=True, indent=2))
    return 1 if result["rejected_records"] else 0


if __name__ == "__main__":
    sys.exit(main())

The header check happens before rows become dictionaries. A missing, reordered, or duplicate column name fails early. Each later record must have exactly three fields, so an extra separator cannot quietly shift the document reference into another column. Quoted commas are handled by the CSV parser.

The duplicate check runs only after a record passes its field checks. It remembers the first record for each exact key, then flags a later repetition. It reports whether the document reference matches. Both kinds of repetition stop this demo import; neither is silently discarded.

Read the report before touching the catalogue

Run:

python3 check_catalog.py catalog.csv

The process exits with status 1 and prints this report:

{
  "records": 7,
  "rejected_records": 3,
  "issues": [
    {
      "record": 5,
      "problems": [
        "model: outside the agreed grammar (U+0410)"
      ]
    },
    {
      "record": 7,
      "problems": [
        "key repeats record 4: different document"
      ]
    },
    {
      "record": 8,
      "problems": [
        "model: surrounding whitespace",
        "model: outside the agreed grammar"
      ]
    }
  ]
}

There are three flagged records, with different decisions to make:

Record	What the checker found	What to ask the source owner
5	The model contains U+0410	Is the Cyrillic character intentional? If so, should this supplier's contract accept it?
7	The key repeats record 4 with another document	Is this a replacement document, a duplicate export, or another product variant missing from the key?
8	The model has a trailing space	Is that space accidental? Would removing it create another copy of record 4?

The repeated key points back to record 4. That does not prove record 4 has the correct document. The file is held as a whole, so the earlier record is not automatically imported as the winner. Keep it held until the source owner resolves the document conflict or confirms a revised product key.

For a positive test, save this separate fixture as clean.csv:

maker_id,model,document
M01,0017,P-0017.pdf
M01,17,P-17.pdf
M01,AX-5,P-AX5.pdf
M02,AX-5,Q-AX5.pdf

python3 check_catalog.py clean.csv exits with status 0 and reports four records with no issues. This fixture tests the passing path. Removing disputed records from the original file would not resolve the supplier's contradictions.

Make the exit status matter

The checker returns 0 when this file contract passes, 1 for flagged records, and 2 when the input cannot be read or its header or CSV parsing fails. Input errors go to standard error; rejected-record reports go to standard output. A scheduled job should stop before its import step on either nonzero result.

For a dry run, this shell wrapper writes the normal report and leaves the import unimplemented:

if python3 check_catalog.py catalog.csv > catalog-report.json; then
    echo "Contract passed. The reviewed import step can follow."
else
    echo "Import held. Read the report and standard error." >&2
fi

Store the received file before validation and run the importer against that same saved copy. A report for yesterday's bytes does not validate a new upload with the same filename. Keep the source version and report together so someone can trace a disputed document assignment later.

The importer must also preserve the text key. Converting the code to an integer after this check would reintroduce the original problem. Check the destination schema and uniqueness rules before connecting the steps.

What was tested, and what still needs review

The local suite passed 17 tests. They cover distinct leading-zero codes, the same model under different manufacturers, both duplicate cases, Cyrillic lookalikes, whitespace, empty references, BOM input, quoted commas, wrong field counts, invalid headers, malformed quoting, control characters, record numbering, and CLI exit statuses. A source hash was unchanged after each fixture run.

Does a passing result validate the document?

No. The checker tests the reference field but never opens the document. A passing result means that the file meets the stated contract. Someone still needs to check which document belongs to each model.

Can the checker restore zeros lost before export?

No. If the received file already says 17, this script cannot tell whether the supplier intended 0017. Ask for a corrected export or another authoritative identifier.

Can the same checker handle a large feed?

The demo keeps unique keys and the issue list in memory, so memory use grows with the input. For large feeds, store keys in a staging table with an appropriate uniqueness rule and stream issues to a report. Keep the file contract explicit.

After the source owner resolves a disputed record, validate the revised file again. Keep that revision with its new report before enabling the product join.

AI assistance was used to draft this tutorial and example code. The examples are synthetic; the code and reported results were run locally.

DEV Community: Igor Spravtsev