A few months ago I decided to build a reusable engine to make financial reconciliation easier. I had already done reconciliation work for companies in West Africa, closing out exports of around 100,000 transactions, and a lot of people in my network work in fintech. The one problem that keeps coming up in those conversations is reconciliation, and how painful it becomes once the data load grows.
So my first move was not to write code. It was to look for a tool that already solved the problem. To my surprise, there was almost nothing usable. The tools I found were either too general, because data reconciliation is a very broad problem, or they targeted finance but had not been maintained in years. Nothing was directed specifically at financial reconciliation and optimized for it. So I decided to take up the challenge. The problem was interesting on its own, and it was also a good way to deepen my knowledge of Go. The result is an open-source engine called Reconify, that can be used as a CLI, and you can find the repository here.
Here is where it landed. The current version reconciles 10 million records per side, 20 million in total, in 95 seconds on an eight-core Apple M1 Pro with 16 GB of RAM. It did not start there. The first design was fast on convenient files and fell apart on realistic ones, so the architecture had to change three times. The rest of this article is how.
Why reconciliation matters
A payment, most of the time has to work out on two sides. Your own ledger says the customer paid, and the provider is supposed to send the money and report it back. Reconciliation is the job of proving that those two records describe the same event, and flagging every case where they do not fully agree.
When they do not agree, real money is missing, a payment is double counted, or a settlement is stuck. So a finance team cannot close the books until the two sides are tied up.
The mechanic is quite simple to state. One side is the left source, usually your internal ledger. The other is the right source, the provider or bank export. The engine pairs a left row with a right row on a shared key, normally a reference like ORD-2026-84921, then classifies the pair:
- Matched: both sides agree on the reference and the amount.
- Mismatched: the reference matches but a field such as the amount does not.
- Unmatched: a row sits on one side with no counterpart on the other.
This is a simple problem on a clean file. It stays easy until the files get large, wide, and inconsistent.
What makes reconciliation hard
Reconciling two financial exports looks simple until the files become large, wide, and slightly inconsistent. A provider may call a field merchant_reference, an internal ledger may call it external_reference, and a settlement file may represent several payments as one group. The engine still has to decide which records correspond, classify differences, preserve duplicate semantics, and explain every result.
I believe a customer should be able to reconcile a large daily export on a small cloud machine without turning the run into an unpredictable memory event. The benchmark therefore measured more than row count. It recorded schema width, input bytes, runtime, peak resident memory, throughput, and result parity.
Why I chose Go
I chose Go deliberately. At the start I assumed most of my advantage would come from concurrency, which turned out not to be true, at least not early on. I also wanted a language I could iterate in quickly, because the engine would go through several rewrites before it was good.
I did weigh the alternative honestly. I do not know Rust, so I vibe coded the same small routine in both Go and Rust and compared them. The gain was only about 10 to 20 percent in Rust's favor. For a language I already know and can write quickly, that trade was easy to accept.
Three iterations, one benchmark
The first design was fast on narrow synthetic files. A realistic benchmark exposed its cost. The engine then moved through three architectural iterations:
- Architecture 1, memory-indexed streaming matching, released before the partitioned backend.
-
Architecture 2, partitioned execution, released in
v0.4.0. -
Architecture 3, parallel partition processing, released in
v0.4.1.
The sections below walk each iteration in order, from the memory-bound first design to that partitioned run, alongside the benchmark that forced every change.
How to measure
Three metrics measure the engine's performance. Each one answers a different operational question, and no single one is enough on its own.
- Runtime is wall-clock time from the start of the run to the completed output. A long runtime means the reconciliation stays active longer, costs more on a usage-priced machine, delays downstream settlement work, and raises the chance of hitting a job timeout. Runtime is the number people notice first, but it is not enough on its own.
- RSS, or resident set size, is the peak amount of the process's memory that stayed in RAM. A high RSS can trigger swapping, an operating-system kill, or a container eviction when it approaches the machine's memory limit. A lower RSS leaves room for the operating system, other services, and temporary buffers. Partitioning reduces this peak, but it spends more time reading and writing temporary files.
-
Throughput is the amount of input processed per second. Here it is calculated as
left rows + right rowsdivided by runtime. Higher throughput usually means the engine finishes a fixed workload sooner, but it can hide a resource trade-off. More workers raise throughput while also increasing RSS, CPU contention, and temporary disk traffic. A very high number from a run that discards output is also not a production guarantee. It has to be read with the output mode, hardware, schema width, and correctness result.
So the goal here is to answer one question: which run finishes within the required time while keeping memory, CPU, and temporary disk usage inside the deployment budget and producing the same correct result?
Architecture v1: fast lookup, expensive memory
For the first engine version, I optimized the inner match operation. The engine streamed the right file into a memory-resident hash index, streamed the left file, looked up each normalized reference, classified the candidate, and wrote the result. A hash lookup is a good fit for exact reference matching, so the design looked strong in early tests.
However, the execution model was not optimal. The right-side index retained rows, reference buckets, duplicate state, and the values needed by the output path. Wide records increased the cost of every one of those structures. So a fast lookup did not make the working set small.
The initial design stayed because it remained the simplest and fastest path for small and medium files. It also provided the reference behavior the parity tests needed. The trade-off was a memory curve tied to the size and shape of the right-side input.
I expected a memory-bound design, but not to hit its ceiling this soon. When the benchmark moved from narrow fixtures to realistic exports, the cost showed up fast.
At 5 million total records, the width fixture held the row count constant and varied only the schema width. The table below shows how runtime and peak RSS climbed as the fixture grew from 15 to 30 columns:
| Schema width | Runtime | Peak RSS |
|---|---|---|
| 15 columns | 21.09 s | 60.8 MB |
| 20 columns | 23.65 s | 67.8 MB |
| 30 columns | 31.62 s | 81.3 MB |
The real-data test surprised me even more than the synthetic benchmark.
The width increase did not merely add parsing time. It increased the amount of data the execution model had to carry while matching. So the next design had to bound the working set before it added concurrency.
Architecture v2: partition the working set
Partitioning came to me while I was thinking about how the engine loads data. If the engine loads one side and compares it against the other, then I do not need the whole side in memory at once. I can load one part of a side, run the reconciliation against it, and move to the next part.
The first idea was to partition one side only. However, that left the other side as one large file, so the problem was only half solved. Partitioning one side leaves two bad choices: load the complete other side into memory, or scan the complete other file once for every partition. So the goal became simple: partition both sides, then optimize the matching, which is the easy part once the working set is bounded. This shape also turned out to be useful for multi-source reconciliation.
So instead of one scan, the engine makes two staging scans. First, it reads the left CSV and writes each row to a left partition file. Then it reads the right CSV and writes each row to a right partition file. At the end of staging, neither complete source is resident in memory.
The engine then processes one partition number at a time. For a reference pass, it opens left-003 and right-003, builds an in-memory index for the right partition, streams the left partition against it, writes a result chunk, and releases that local state. Grouped passes sort and merge the two partition files while keeping only the current group.
Staging both sides is what lets the engine pair partition 003 with partition 003 and scan each source exactly once.
The partition selector uses the configured matching or grouping column. The staging hash applies the same trimmed key rule on both sides, so a candidate for reference ORD-2026-84921 routes to the same partition in each source. This routing rule is a correctness requirement, not a claim that all candidates load together at once.
The partitioned backend stages both CSV inputs into temporary partition files. It then processes one matching partition pair at a time:
- The parser normalizes the configured key.
- A hash selects the partition.
- The engine builds a local index or an external grouped sort for that partition.
- The partition emits typed results and carry-forward metadata.
- The temporary working set is released before the next partition.
For grouped reconciliation, external sort and merge-read keep only the current left and right groups in memory. The cost moves to temporary disk and extra passes over the data. That trade-off is deliberate. Disk is slower than memory, but it is easier to budget than an unbounded process-wide index.
Partitioning reduced peak memory, but serial partition processing left CPU capacity unused. It also introduced temporary files, cleanup paths, sort failures, queue design, and skew. So the new architecture solved the memory problem and created independent work units at the same time. Those independent units are what made parallelism possible, and this is where Go became useful.
Architecture v3: parallel partitions with backpressure
Parallelism before partitioning would have multiplied the memory-bound design. Each worker would need its own index, or every worker would contend for a global one. Either choice would make the original failure mode worse.
Partitioning changed the shape of the problem. Each partition became a bounded unit with explicit input files, a local index or grouped merge, a typed result chunk, and a summary. So the scheduler could assign independent partitions to workers while keeping the final writer single-owned.
For the parallelism, I started with a worker pattern where each worker reconciled and sent its results straight to the result writer. The idea sounds fine, but it broke quickly. When many workers produce results faster than a single writer can consume them, the writer becomes the bottleneck, and in my first version the writing was very slow. That is where the writer crashed under load.
Adding a plain queue does not solve the problem on its own. The queue is a backpressure mechanism, not a second result store. If the writer is slower than the workers, the bounded queue fills and the workers pause. Without that boundary, parallelism would recreate the memory problem by accumulating completed results in process memory.
Watching the writer stall against the disk sent me reading about how a printer handles the same problem. A printer cannot accept data as fast as a program produces it, so the operating system spools the work to disk and feeds it out in order. That design is called SPOOL, Simultaneous Peripheral Operations On-Line, and it is exactly the shape I needed. So the split idea came from there.
So workers do not send every result event through a channel. Using the spooling design, each worker writes its typed events to a private disk-backed chunk and sends only a small descriptor to a bounded queue. The descriptor holds the partition number, the chunk path, the summary, and the carry-forward metadata.
What parallelism accelerates, and what it does not
The parser remains a sequential stream for now. --partition-workers starts after staging has already read the source files, so adding workers does not make one CSV file parse in parallel. The release comparison shows only a small parse change, from 2.14 seconds in v0.3.0 to 1.85 seconds in v0.4.0 and 1.83 seconds in v0.4.1 for the same 500,000-row left file. The parallel feature is not a parser optimization.
The matching stage is different. On a fixed 1 million-record, 20-column fixture, the partitioned single-source run moved from 24.41 seconds with serial workers to 19.94 seconds with four workers. A three-source run moved from 23.85 seconds to 19.65 seconds. The measured speedups were 18.3 percent and 17.6 percent. The result does not support a claim that multi-source is always faster with parallelism. It supports a narrower conclusion: independent partitions improve the reconciliation stage, while parsing stays outside the parallel region.
The memory trade-off is visible too. Single-source peak RSS rose from 36.7 MiB to 108.0 MiB with four workers. Three-source peak RSS rose from 34.7 MiB to 57.6 MiB. The queue bounds completed descriptors, but each active worker still owns staging, indexes, sort buffers, or result chunks. So more workers trade elapsed time for memory and temporary I/O.
Comparing the releases
To compare the improvements on realistic data, I built a small project called Paybridge to generate files closer to the transactions I handled on past projects. These fixtures are not official provider exports, and they do not represent every customer schema. The measured records include both sides of each reconciliation, so 20 million total records means 10 million left rows and 10 million right rows. Grouped passes behave differently from exact reference matching. Filesystem speed, cache state, CPU count, schema width, key skew, and output format all affect the result.
I built the same v0.3.0, v0.4.0, and v0.4.1 binaries and ran each against the same 1 million-record, 20-column clean dataset. Each run used the memory backend, NDJSON output discarded at the process boundary, and the same Apple M1 Pro host.
| Release | Parse 500k rows | Reconcile 1M records | Reconcile peak RSS |
|---|---|---|---|
v0.3.0 |
2.14 s | 5.74 s | 726 MiB |
v0.4.0 |
1.85 s | 4.22 s | 723 MiB |
v0.4.1 |
1.83 s | 4.15 s | 705 MiB |
The parser improves before parallelism exists because other parser and allocation changes landed earlier. Parse time falls 14.5 percent from v0.3.0 to v0.4.1. Reconciliation time falls 27.7 percent over the same comparison, while v0.4.1 is only 1.7 percent faster than v0.4.0 on this memory-backed single-pair fixture. Its main parallel feature is not active in this comparison. The larger architectural gain appears when the partitioned backend is enabled, where the working set is bounded and partitions become schedulable.
Performance means nothing without correctness
A fast reconciliation that returns the wrong answer is worse than a slow one, because someone downstream trusts it. So every architecture change had to clear the same bar before it was allowed to be faster: produce the same result as the simple memory design, on every fixture.
The engine checks this with a parity suite that compares more than summary counts. It compares row-level events, matched counterpart ownership, grouped members, anomaly classifications, duplicate annotations, and the final summary between the memory backend and the partitioned backend. A partition that reorders work is only correct if it lands on identical output.
What I am still working on
The engine is not finished, and the parser is the next target. The staging scan is sequential, and it can skew badly on uneven data. Three ideas remain on the list:
- Bloom-filter pruning over left keys, so a right row with no possible match is dropped before it is ever staged.
- Adaptive broadcast joins, so a small side that fits the memory budget skips partitioning and runs in a single pass.
- Binary staging, so the engine stops parsing each CSV twice.
Each one attacks a different bottleneck, and they will probably become the next article.
Final thoughts
The most useful result of this project was not the final runtime. It was learning that the design that looked fastest on convenient data was not the design I wanted to run in production.
Go turned out to be a good fit for that path. Once partitioning had carved the work into independent units, goroutines, channels, and a bounded queue turned divide-and-conquer into a scheduling problem rather than a memory gamble.
The next steps are the three changes above, plus more benchmarks and correctness tests, since parallelism raises the risk of race conditions.
The engine is open source. You can read the code, the release history, and the benchmark setup in the Reconify repository. If it is useful to you, a star helps other teams working on reconciliation find it.
Top comments (0)