I added 10 validation rules to an e-invoice parser without changing one existing warning. Here is the test setup

The validation output of my e-invoice parser had a design flaw: it was a list of English sentences.

"validationWarnings": ["Tax basis + VAT = 1728.70, but grand total is 1729 (BR-CO-15)"]

Fine for a human. Bad for an accounts-payable workflow that wants to block on real errors, only log hints, and show the user which field is wrong. The only way to do that was to parse the strings. And once output is published, it is an API, whether you like it or not.

So this week's job was: add machine-readable findings and more rules without changing a single existing string, because any workflow built on the old output has to keep working.

Disclosure: I built the XRechnung & ZUGFeRD Parser on Apify. This post is about the testing approach, which works for any parser.

The new shape

Every finding now has a rule ID, a severity and the business term it refers to. The message is exactly the old string.

"validationFindings": [
  {"rule": "BR-04",    "severity": "error", "field": "BT-3",   "message": "Missing required field BT-3 type code"},
  {"rule": "BR-CO-15", "severity": "error", "field": "BT-112", "message": "Tax basis + VAT = 1728.70, but grand total is 1729 (BR-CO-15)"},
  {"rule": "BR-CO-16", "severity": "error", "field": "BT-115", "message": "Grand total - prepaid + rounding = 729.30, but amount due is 729 (BR-CO-16)"}
],
"errorCount": 3,
"warningCount": 0

That is real output for a UBL credit-note sample from the Mustang project's test files. validationWarnings is still there and still a list of strings.

The safety net: old code vs new code on public invoices

I did not write new unit tests first. I froze the old module and compared it with the new one on real files, a golden-master test:

import glob, importlib.util, os, sys

def load(name, path):
    spec = importlib.util.spec_from_file_location(name, path)
    mod = importlib.util.module_from_spec(spec)
    spec.loader.exec_module(mod)
    return mod

old, new = load("old", sys.argv[1]), load("new", sys.argv[2])
bad = 0
for path in sorted(glob.glob(os.path.join(sys.argv[3], "*"))):
    if not path.endswith((".xml", ".pdf")):
        continue
    data = open(path, "rb").read()
    o, n = old.parse(data), new.parse(data)

    # 1. every old warning must still be there, same text, same order
    old_w, new_w = o["validationWarnings"], n["validationWarnings"]
    prefix_ok = new_w[: len(old_w)] == old_w

    # 2. every other old field must be unchanged
    diff = [k for k in o
            if k not in ("validationWarnings", "validationWarningCount") and o[k] != n.get(k)]

    if not prefix_ok or diff:
        bad += 1
        print(os.path.basename(path), "prefix_ok =", prefix_ok, "diff =", diff)
print("files with differences:", bad)

Two rules do the work:

1. Old warnings are a prefix of the new list. New checks may only append. This forces the code structure too: the historical checks stay in one block in their original order, new checks come after it.
2. Every old field is equal. New fields are allowed, changed ones are not.

The test data is public

No customer invoices needed. Two open sources cover most real-world shapes:

• the KoSIT XRechnung test suite: 66 standard cases in UBL and CII syntax (invoices, corrections, self-billing),
• sample files from the Mustang project: 7 files including a ZUGFeRD PDF, a Factur-X MINIMUM file and a UBL credit note.

What the run showed

Result on 73 public files	Files
Old output identical, no findings	70
Old output identical, findings unchanged (now with rule IDs)	2
Old output differs	1

The one difference was a good catch. For CII invoices I had added contactDepartment to the existing seller object, because rule BR-DE-5 accepts a department name as contact point. A new key inside an existing object is still a diff. It is additive, so I kept it, but I only knew about it because the script compares whole objects and not just the fields I had in mind.

The two files with findings:

File	Findings
Mustang ubl-creditnote.xml	BR-04, BR-CO-15, BR-CO-16 (all error)
Mustang facturFrMinimum.xml	BR-CO-16 (warning)

All 66 KoSIT standard cases produce zero findings, which is what you want from a rule set: valid invoices stay quiet.

Quiet files prove nothing about new rules though, so I derived four variants from one KoSIT case, three of them deliberately broken:

Own test file	New findings
Credit note (381) without reference	FD-PRECEDING (warning)
Credit note (381) with reference	none, precedingInvoice = {number: "123456XX", date: "2016-04-04"}
Deliberately broken XRechnung (UBL)	BR-CO-16, BR-DE-6, BR-DE-9, BR-DE-23-a
XRechnung CII without seller contact	BR-DE-2

Three decisions that mattered more than the code

Severity depends on the profile. A Factur-X MINIMUM file is not a full EN 16931 invoice by design. Reporting its missing fields as errors would make every such file "broken". So missing EN fields are errors for full profiles and warnings for MINIMUM and BASIC WL. That is why the same rule (BR-CO-16) appears as error in one row above and as warning in the other.

Never invent an official rule number. The IDs were checked against the public KoSIT and EN 16931 Schematron files. For my own plausibility checks, such as "credit note without a preceding invoice reference", I use a private prefix (FD-...) instead of borrowing a BR number that says something slightly different.

Say what it is not. This is a set of plausibility checks, not a Schematron validation. An invoice with errorCount: 0 can still be rejected by the official validator. That sentence is now in the README, next to the rule table.

One gap the test did not close

On the CII file without a contact, BR-DE-7 (seller e-mail) stays silent: the parser has always filled the e-mail from the electronic address when its scheme is EM. Changing that would alter an existing field, so rule 2 of the test says no. It is in the backlog as a documented limitation instead of a silent behaviour change.

Takeaways

• Published strings are an API. Freeze them and add structure next to them.
• A short old-vs-new script over public sample files finds what you did not think of.
• Valid files must stay quiet; you need deliberately broken files to see new rules fire.

Try it

The parser turns XRechnung (UBL/CII) and ZUGFeRD/Factur-X PDFs into JSON with the findings above, for $0.003 per invoice: XRechnung & ZUGFeRD Parser on Apify.

Question: how do you keep old output stable when you extend a parser or an API: golden files, contract tests, something else? And have you been bitten by "additive" changes that still broke a consumer?