How I test a bank statement converter: read every export back with hledger, beancount and ofxtools

I build a Mac app called Balanced that converts bank and card statements (text PDFs, scans, CSV) into CSV, Excel, OFX/QFX/QBO, QIF, YNAB, Actual, Xero, Beancount and Ledger files. Writing the parser was the easy half. This post is about the other half: how I check that the files it produces are actually right, and not just files that open.

A file that opens is not a file that is right

The first version of my tests only checked that each exporter produced something and that it was well formed. That passes happily while the numbers are wrong. A converter's whole job is that a number in the output is a number that was on the statement, so the tests have to prove that, with tools that have no stake in my code.

The totals check

Before any export, the app checks the statement against itself: opening balance plus all transactions must equal the closing balance, and every running balance column has to follow from the one before it. If it doesn't, you see a mismatch in the preview instead of importing it. The same check doubles as a test oracle: if a real sample statement doesn't balance after conversion, either the parser or the statement layout needs attention.

Read every export back with an independent tool

For each format I use a reader I did not write:

• OFX/QFX/QBO: ofxtools parses the file and I compare transaction count, dates, amounts and the ledger balance to the source.
• QIF: quiffen reads it back.
• Beancount: bean-check has to accept it, including the balance assertion for the closing balance.
• Ledger and hledger: ledger and hledger check must load it, and the balance report has to match the statement.
• Excel: openpyxl reads the cells back, so I know amounts are real numbers and not text.

That last-mile read-back caught things my own tests had called fine. The best example: Beancount refused a file because the closing balance assertion didn't hold, which meant my exporter and my parser disagreed by one posting. A hand-written comparison would not have found it.

Real statements, many regions

The tests run over 83 real public sample statements, from many banks and countries (DD/MM dates, decimal commas, apostrophe and lakh digit grouping, scans). The conversion output is also byte-identical when run under six different macOS regions, which matters because a Mac with a German or Indian locale must not change how 1.234,56 is read or written.

What it is still bad at

Layouts it hasn't seen may need a column fix in the preview (it remembers the fix per bank), and poor phone photos of statements can still misread digits. The totals check usually catches those.

If you want to try it

It is free to convert and check everything; the free version exports the first 10 lines per file. Pro is USD 59 per year or 7.99 per month. I'd like to hear which statements it reads wrong: https://kayabyte.dev/mac/balanced/?ref=devto-bal

(Disclosure: I'm the developer.)